Download the Book (PDF): Introduction A consultation is a piece of work that two people accomplish out loud. A patient arrives with something wrong, or something they fear is wrong. A clinician has perhaps ten or fifteen minutes to find out what it is, decide what it means, and agree with the patient on what happens next. Almost all of that work is done in talk: in questions and answers, in stories begun and interrupted, in pauses, in the way a finding is announced and the way it is received. Medicine has powerful instruments, but in the ordinary consultation its most frequently used instrument is conversation. For most of the history of medicine, that conversation left no trace except the clinician's note, which recorded what the doctor judged relevant and nothing of how it came to be said. When researchers began, in the 1960s and 1970s, to place tape recorders on consulting-room desks, they found that the encounter they had assumed they understood was stranger and more consequential than anyone had noticed. Patients were being cut off within seconds of starting to speak. Questions that seemed neutral were steering answers in predictable directions. Diagnoses were delivered in forms that invited or foreclosed discussion. Concerns that patients had come in specifically to raise went unmentioned from the first word to the last. None of this was visible in the notes, and very little of it was visible to the participants themselves. This booklet is about the body of qualitative research that grew from those recordings, and about the methods it uses. Its subject is discourse analysis applied to medical consultations: the close, systematic study of recorded talk and embodied conduct between clinicians and patients. It is written for readers who want to understand what that research has found and how it is done: clinicians curious about the evidence behind communication training, students and researchers in health sciences, linguistics and sociology who are planning their own studies, and anyone who has left a consultation with the uneasy sense that something important was not said and wondered why. The controlling argument The argument of the booklet can be put in a sentence. The authority of medicine and the accuracy of diagnosis are both produced, turn by turn, in the fine structure of talk, and so the gaps that harm patients are usually built into the shape of the interaction rather than into anyone's intentions or character. That claim has two halves, and both matter. The first is about power. It is common to talk about the doctor-patient relationship as if power were a fixed quantity that one party possesses and the other lacks. Recorded consultations show something more interesting: power is exercised through particular conversational positions and practices, such as who gets to ask the questions, who decides when a topic is closed, and how a diagnosis is framed as a verdict or as a conclusion from evidence the patient can inspect. These practices can be described precisely, and once described they can be changed. The second half is about diagnosis. A consultation is not only a social ritual laid on top of a technical procedure. The information on which clinical reasoning depends has to pass through the conversation, and the conversation's structure shapes what gets through. A question designed to expect "no" tends to receive "no". A patient who has been interrupted after eleven seconds may never reach the symptom that mattered. At the same time, the way patients talk can itself be diagnostic evidence: researchers have shown that the conversational profiles of people with epilepsy differ from those with functional seizures, and that people with neurodegenerative memory disorders talk about their memory differently from those whose memory complaints are functional. Talk is both the channel through which diagnostic information travels and, sometimes, the information itself. The practical consequence is the one that gives the field its value. If communication failures were mainly failures of goodwill, the only remedy would be exhortation: be kinder, listen more. Because many of them turn out to be features of sequence and design, they can be addressed with small, specific, testable changes. One of the best-known studies in the field found that altering a single word in a doctor's closing question substantially reduced the number of patients who left with concerns they had never raised. That result would have been impossible to obtain without recordings and without a method for analysing them. What the booklet covers, and what it leaves out The chapters move from method to findings to application. The first two establish how the field works: where the study of recorded consultations came from, what the main analytic traditions are and how they differ, and the practical craft of collecting, transcribing and analysing audio and video. Readers planning a study will find the ethical, technical and analytic decisions set out in some detail, because these are the decisions that determine whether a project produces anything worth reading. The next four chapters follow the consultation through its natural course. Chapter 3 examines turn-taking and the overall architecture of the visit, including what happens in the opening seconds. Chapter 4 looks at clinicians' questions and at the long debate over the asymmetry of medical talk. Chapter 5 turns the analysis around to ask how patients raise concerns, ask questions and pursue their own agendas in a setting that does not make it easy. Chapter 6 examines how diagnoses and bad news are delivered and received, and Chapter 7 how treatment recommendations are made, resisted and negotiated. The final two chapters address consequences. Chapter 8 gathers the evidence on communication gaps: unvoiced agendas, unmet concerns, misunderstandings in multilingual consultations, the special difficulties of interpreted encounters, and the ways in which patients' age, language and background shape how they are questioned and heard. Chapter 9 describes how findings from recorded consultations have been turned into interventions, training methods and diagnostic aids, and what the method's limits are. Some things are deliberately left out. The booklet does not attempt a survey of every theory of communication in medicine, nor a review of the large quantitative literature correlating communication scores with satisfaction and adherence, though that literature is mentioned where it bears on the argument. It concentrates on the qualitative traditions that work directly with recordings, above all conversation analysis, which has produced the most cumulative and most practically consequential body of findings, and on the neighbouring traditions of interactional sociolinguistics and critical discourse analysis where they add something distinctive. Most of the research described comes from primary care and outpatient specialties in English-speaking and northern European countries, because that is where most of the work has been done; the booklet notes where this matters. A note on examples Conversation analysis has a strong convention of showing the data: presenting transcribed fragments so that readers can check the analyst's claims against what was actually said. The published literature contains thousands of such fragments, collected under ethical agreements that restrict their reproduction. Rather than reproduce them, this booklet describes published findings in prose and, where a short illustration helps, uses brief constructed exchanges that are clearly marked as illustrative. They are built to display a practice that the cited research has documented, not to stand in as data. Readers who want to see the real recordings in transcript should go to the studies themselves, listed in the Notes and Further Reading, where the evidence is set out in full. Why this matters now Consultations are changing. Electronic records have put a screen between doctor and patient; telephone and video consultations have removed much of the embodied information both parties once relied on; and ambient recording systems that transcribe and summarise consultations automatically are spreading quickly through clinical practice. Each of these changes alters the structure of the interaction, sometimes in ways that no one designing the technology anticipated. The methods described here are the best tools we have for finding out what those changes do. They are also, increasingly, tools that clinicians and patients can use themselves, since the recording of consultations is no longer the preserve of researchers. The discipline that emerged from a tape recorder on a consulting-room desk has become one of the more practically useful branches of the social sciences. It deserves to be better known, and it rewards being understood from the inside. Chapter 1: From the Tape Recorder to a Science of Consultations The systematic study of recorded medical talk began with a practical worry. In 1970s Britain, two researchers, Patrick Byrne and Barrie Long, collected well over a thousand audio recordings of general practice consultations, most of them made by the doctors themselves. Their report, published in 1976 as Doctors Talking to Patients, did two things that shaped everything afterwards. It proposed that consultations had a recognisable order of phases, from establishing a relationship through discovering the reason for attendance, examining, considering the condition, and detailing treatment, to termination. And it showed that doctors had characteristic styles, ranging from strongly doctor-centred to strongly patient-centred, that tended to persist whatever the patient in front of them needed. Many doctors, listening to their own tapes, were surprised by what they heard. That surprise is the founding experience of the field. Clinicians, like everyone else, have poor insight into the moment-by-moment detail of their own talk. They remember what they meant, not what they said, and they remember the gist of a patient's answer, not the hesitation that preceded it. Recordings make the detail available for inspection. The question every subsequent tradition has had to answer is what to do with that detail once it is available. Counting: interaction analysis and coding systems The first answer was to count. Social psychologists had already developed methods for coding small-group interaction into categories, most influentially Robert Bales's Interaction Process Analysis of 1950, which classified each utterance as, for example, giving information, asking for an opinion, or showing tension. Applied to medical encounters, coding offered comparability, statistical power and an obvious route to linking communication with outcomes. The most widely used system in medicine is the Roter Interaction Analysis System, known as RIAS, developed by Debra Roter from the 1970s onwards. RIAS divides talk into utterances, the smallest units expressing a complete thought, and assigns each to one of several dozen categories grouped broadly into task-focused talk (giving and asking for biomedical or lifestyle information, counselling) and socio-emotional talk (empathy, reassurance, agreement, social chitchat). Coders work directly from the audio rather than from transcripts, which makes the system relatively fast. Because the categories are standardised, studies can be pooled, and RIAS has been used in hundreds of studies across many countries and languages. Coding has produced real findings. It documented that physicians typically do most of the talking and ask most of the questions; that patient-centred talk is associated with higher satisfaction; and, in a study by Wendy Levinson and colleagues published in 1997, that primary care physicians who had never been sued differed in measurable ways from those who had, spending more time in routine visits, using more orienting statements about what would happen next, and using more humour and facilitative talk. Lisa Cooper's group used RIAS to show that visits between patients and physicians of the same race were longer and rated more positively, and that Black patients in some samples experienced less patient-centred communication than white patients. The limitation of coding is built into its strength. To count an utterance as a "question" or a "reassurance", the coder must decide in advance what category it belongs to and must treat all members of a category as equivalent. But the meaning of an utterance depends heavily on where it occurs and how it is designed. "Anything else?" at the start of a visit and "anything else?" as the doctor stands up to leave are, in the coding scheme, the same act. In the interaction they are very different, and patients respond to them differently. A coding system can detect that a doctor asked many closed questions; it cannot show how those questions constrained the answers, or what a patient did to get a concern onto the agenda despite them. Interpreting: the voice of the lifeworld A second answer came from sociolinguistics and critical theory. Elliot Mishler's The Discourse of Medicine, published in 1984, is the classic statement. Mishler analysed transcripts of internal medicine interviews and argued that two "voices" contended in them. The voice of medicine speaks in the technical, decontextualised terms of symptoms, durations and locations. The voice of the lifeworld speaks in terms of the patient's everyday experience: work, family, fear, the meaning of illness in a particular life. In the typical interview, Mishler argued, the physician's questions repeatedly interrupted and suppressed the voice of the lifeworld, returning the patient to the categories medicine needed. The unit that did this work was a three-part cycle: physician question, patient response, physician assessment or next question. Control of the first and third positions gave the physician control of the whole. Mishler's framework was taken up by researchers who wanted to connect the micro-detail of talk with larger structures of institutional power. Howard Waitzkin's The Politics of Medical Encounters (1991) examined how consultations reproduced ideological assumptions about work, gender and ageing, often by treating social problems as individual medical ones. Norman Fairclough's critical discourse analysis, set out in Discourse and Social Change (1992), used medical interviews as a central example of how institutional discourse types are mixed and contested, contrasting a "standard" medical interview with an alternative, more conversational form. Christine Barry, Nicky Britten and colleagues, in a study of general practice consultations published in 2001, applied Mishler's categories and found that consultations in which both parties spoke in the voice of the lifeworld, or in which doctors responded to patients' lifeworld contributions, had better outcomes by several measures than those in which the lifeworld was ignored or suppressed. This interpretive tradition has the virtue of asking big questions about power, ideology and meaning. Its weakness, critics argued, was that it sometimes read those structures into the talk rather than out of it. If the analyst already believes that medicine suppresses the lifeworld, every interruption can be seen as suppression. What was needed was a method that could show, from the participants' own conduct, what an utterance was doing. Describing: conversation analysis That method was conversation analysis, which emerged in California in the 1960s from the work of Harvey Sacks, Emanuel Schegloff and Gail Jefferson. Its founding insight was that ordinary conversation is orderly at a level of detail no one had thought to examine, and that its order is produced by the participants themselves, who display to each other, turn by turn, how they have understood what came before. The key evidence for what an utterance does is what the next speaker does with it. If a doctor says "the chest sounds clear" and the patient responds "oh good", the patient has treated it as good news; if the patient responds "so why am I still coughing?", the patient has treated it as incomplete. The analyst's claims are anchored in these displayed understandings rather than in categories imposed from outside. Conversation analysis was first applied to medicine in the 1970s and 1980s by researchers such as Richard Frankel, Candace West, Christian Heath and Paul ten Have. By the 1990s a coherent program had emerged, stimulated by the collection Talk at Work, edited by Paul Drew and John Heritage in 1992, which set out how institutional talk differs from ordinary conversation. Institutional interactions, they argued, are marked by goal orientations tied to institutional identities, by special constraints on what counts as an allowable contribution, and by distinctive inferential frameworks. A doctor's question about alcohol is heard differently from a friend's. The contrast with ordinary conversation became a working tool: by identifying what is special about medical talk, the analyst can locate where its peculiar pressures lie. The major synthesis came in 2006 with Communication in Medical Care, edited by Heritage and Douglas Maynard, which assembled studies of each phase of the primary care visit. It was the product of a research program centred at UCLA and several other institutions, and it established conversation analysis as the dominant qualitative approach to the medical encounter. Its distinctive contribution was to link the findings of detailed qualitative analysis to measurable outcomes. Once a practice had been identified qualitatively, such as a particular way of opening a visit or of commenting during a physical examination, it could be coded reliably across large samples and tested for its association with satisfaction, prescribing or unmet concerns. Several of the practices were then tested experimentally. This combination of qualitative discovery and quantitative confirmation is what gives the conversation analytic literature its unusual practical authority. It is also why this booklet draws on it more than on other traditions. Neighbours: interactional sociolinguistics and thematic approaches Two other traditions deserve mention because they answer questions conversation analysis tends to leave aside. Interactional sociolinguistics, associated with John Gumperz, studies how people from different linguistic and cultural backgrounds signal and interpret meaning through "contextualisation cues": intonation, rhythm, choice of words and code, and the conventions that tell a listener how an utterance is meant. It is particularly valuable for multilingual consultations, where misunderstandings arise not from vocabulary but from mismatched expectations about how a story should be told or how a complaint should be framed. Celia Roberts and Srikant Sarangi, working in British general practice, used this approach to show how patients with limited English and doctors trained in different traditions talked past each other in ways that neither could identify at the time. Many health researchers also analyse consultation transcripts using thematic analysis, a flexible approach set out influentially by Virginia Braun and Victoria Clarke in 2006. Thematic analysis identifies patterns of meaning across a data set and is well suited to questions about what topics are raised, what concerns recur, and how participants describe their experience. Its limitation for recordings of interaction is that it usually treats the transcript as a record of content rather than a record of action. It can show that patients mention cost frequently; it is less able to show how cost gets raised, by whom, and with what consequences for the next turn. Choosing an approach These traditions are not rivals so much as tools with different edges. The choice depends on the question. If the aim is to compare large samples of consultations on standardised dimensions, coding is appropriate. If the aim is to understand how a practice works in the interaction, how it is recognised and responded to, conversation analysis is the natural choice. If the question concerns institutional ideology or the reproduction of social inequality, critical discourse analysis offers a vocabulary for it, though its claims need anchoring in the data. Where the problem is cross-cultural misunderstanding, interactional sociolinguistics is often the most revealing lens. Table 1 sets out the main differences. Table 1. Main approaches to analysing recorded consultations. Approach Unit of analysis Typical data Characteristic question Main limitation Interaction coding (e.g. RIAS) Utterance assigned to a category Audio, coded directly How much of each kind of talk occurs, and does it predict outcomes? Treats utterances as context-free Conversation analysis Action within a sequence Audio or video with detailed transcripts How is this action done, recognised and responded to? Slow; small collections; bounded by what is recorded Critical discourse analysis Text in its institutional and social context Transcripts plus documents How does talk reproduce or contest institutional power? Risk of reading theory into data Interactional sociolinguistics Contextualisation cues and inferences Recordings plus participant interviews Why do speakers from different backgrounds misunderstand each other? Heavily interpretive; hard to scale Thematic analysis Pattern of meaning across a data set Transcripts, often content-level What topics and concerns recur? Loses the sequential detail of action In practice many studies combine approaches. A common and productive design begins with conversation analysis to identify a practice, develops a coding scheme from the qualitative findings, and applies it to a larger sample to test associations with outcomes. Another pairs recordings with interviews, asking participants afterwards what they were trying to do, although the conversation analytic tradition treats such accounts cautiously, as reconstructions rather than windows onto what happened. How claims are warranted Because conversation analysis supplies most of the findings discussed in this booklet, it helps to be clear about how its claims are supported, since the logic is unfamiliar to readers trained in experimental or survey research. The first source of evidence is the next turn. Every contribution to a conversation displays its speaker's understanding of the one before, and so offers the analyst, and the other participants, a check on what that earlier turn was taken to be doing. Sacks and his colleagues called this the "next-turn proof procedure". If an analyst claims that a doctor's utterance was heard as a diagnosis, the claim is strengthened when the patient responds as one responds to a diagnosis, and weakened when the patient responds as if to a passing remark. The second source is deviant cases. Once a pattern has been identified across a collection, the analyst looks closely at the instances that do not fit. Often these turn out to confirm the pattern, because the participants themselves treat the departure as noticeable: they pause, apologise, account for it, or repair it. A patient who answers a yes-no question with a long story, for example, may preface the story with a marker that shows awareness that more than a yes or no was due. Such cases show that the pattern is a norm the participants orient to, not merely a regularity the analyst has counted. The third source is distribution. A practice identified in one consultation is interesting; one found across dozens of consultations, with different doctors and patients, in a consistent position and with consistent consequences, is a finding. Conversation analytic studies are typically built on collections of cases, sometimes a few dozen and sometimes several hundred, drawn from a larger corpus of recordings. Finally, and increasingly, claims are tested outside the qualitative analysis altogether. When a practice is well enough described to be coded reliably, its association with outcomes can be measured, and in some cases it can be manipulated experimentally. This is how some of the most useful findings in the field have been established, and it answers the charge, sometimes made against qualitative work, that it produces descriptions without consequences. None of this makes conversation analysis immune to error. Analysts can over-read a single case, can miss what is happening off-microphone or off-camera, and can mistake a feature of a particular clinic's routine for a general property of medical talk. The field's safeguards are its insistence on showing the data, its collaborative habits of analysis, and its gradual accumulation of findings across settings and languages. Studies from Finland, the Netherlands, Norway, the United Kingdom, the United States, Japan and elsewhere have found both striking similarities in how consultations are organised and important local differences, for instance in how much patients are expected to contribute to decisions or how directly diagnoses are stated. What the recordings made visible The most important thing to take from this history is not the differences between traditions but what they share: the conviction that the consultation is an object worth studying in its own right. Before the tape recorder, communication in medicine was treated as a matter of manner, the "bedside" part of bedside manner, separable from the clinical work. The recordings made that separation impossible to sustain. They showed that how a doctor asks a question determines what the patient says, that the patient's answer shapes the diagnosis, and that the form in which a diagnosis is delivered shapes whether the patient accepts it and what they do next. Communication is not the wrapping of clinical care. For large parts of the consultation, it is the clinical care. The chapters that follow depend on that conviction. But before examining what the recordings show, it is worth understanding how they are made and analysed, since every finding in the field is only as good as the data and the craft behind it. Chapter 2: Recording, Transcribing and Analysing Consultations Every finding in this field rests on a recording, a transcript and an analysis, and each of the three involves choices that determine what can later be seen. A study that places a single audio recorder on the desk will capture the words but miss the doctor turning to the screen at the moment the patient begins a difficult disclosure. A transcript that tidies speech into grammatical sentences will lose the pause, the restart and the overlap that show where the interaction strained. An analysis that begins with a hypothesis about power may find power everywhere and nothing else. This chapter sets out the craft of doing the work well. Ethics and consent Recording medical consultations raises ethical questions that go beyond ordinary research consent. The patient is often unwell, anxious or in pain; the consultation may involve intimate examination or disclosure; and the relationship between patient and clinician involves an asymmetry that can make refusal feel difficult. Researchers therefore need to design recruitment so that consent is genuinely free, which usually means that someone other than the treating clinician approaches the patient, that declining has no effect on care, and that patients can withdraw during or after the consultation, including by asking for the recording to be destroyed. A review of the evidence and regulations on video-based research in healthcare communication, published by Ruth Parry, Marco Pino, Christina Faull and Luke Feathers in 2016, found that the available empirical studies generally reported that most patients approached were willing to be recorded, including in sensitive settings such as palliative care, and that participants seldom reported recordings as distressing. The review also set out recommendations: consent should be staged, with agreement to be recorded obtained before the consultation and agreement to particular uses of the recording, such as showing clips in teaching, confirmed afterwards; participants should be able to choose which uses they accept; and researchers should plan for the unexpected, such as a family member arriving who has not consented. Professional bodies publish their own guidance. In the United Kingdom, for instance, the General Medical Council has issued guidance on making and using visual and audio recordings of patients, which distinguishes recordings made for care from those made for research and teaching. Consent to record is also consent to a particular data management regime. Video is identifiable in ways that transcripts are not, and a face or a voice cannot be fully anonymised without destroying much of its analytic value. Most studies therefore distinguish between raw recordings, held securely with restricted access, and anonymised transcripts, in which names, places and other identifying details are replaced with pseudonyms. Where still images or clips are published, specific consent is required, and many projects now use line drawings traced from video frames rather than photographs. Does recording change the consultation? The obvious objection to recording is that it changes the thing being recorded. Clinicians might behave more carefully; patients might hold back. This "observer effect" is real but, the evidence suggests, smaller and shorter-lived than people expect. Participants typically orient to the camera at the start of the encounter and rarely thereafter; the business of the consultation quickly takes over. Studies that have compared recorded and unrecorded consultations on measures such as length or content have generally found modest differences. Two observations help put the concern in proportion. First, many of the practices of interest in conversation analysis, such as the timing of turns, the grammatical design of questions and the placement of responses, are largely beyond conscious control. A doctor trying to seem patient-centred may say more sympathetic things, but is unlikely to change how quickly they take the floor after a patient pauses. Second, what matters for most analyses is the structure of practices rather than their frequency. If recording makes a doctor slightly more likely to ask about the patient's concerns, the way the question is designed and answered is still available for study. The analyst should nevertheless look for signs that participants are orienting to the recording, such as glances at the camera, jokes about being filmed, or explicit remarks, and treat those moments with care. Audio or video? Early studies used audio alone, and audio remains adequate for many questions. Telephone consultations, of course, have no visual channel to lose. But in face-to-face consultations a great deal of the interaction is embodied. Christian Heath's studies of British general practice in the 1980s, collected in Body Movement and Speech in Medical Interaction (1986), showed how gaze, posture and gesture coordinate with talk. A patient who begins to speak while the doctor is reading the notes may restart or pause until the doctor looks up. A doctor may signal the end of a phase of the visit by turning back to the desk before saying anything at all. Jeffrey Robinson's 1998 study of consultation openings showed that patients time the delivery of their presenting concern to the doctor's disengagement from the records and reorientation of gaze and body towards them. Since the spread of electronic health records, these observations have become more pressing. The screen is now a third participant in many consultations, and much of what happens in the room is organised around it: doctors typing while patients speak, turning the screen to show results, or falling silent while they navigate. Audio alone cannot capture any of this. For face-to-face consultations, therefore, video is now the default in most interactional research, ideally with two cameras, one capturing the patient's face and one the clinician's, or a single wide shot that includes both bodies, the desk and the screen. Practical details matter. The microphone should be close enough to capture quiet speech clearly, because quiet speech is often where the important things are said. Recordings should be time-stamped and, where multiple devices are used, synchronised. Screen activity can be captured with screen-recording software, with appropriate consent. For video consultations, recording the call itself captures what each party actually sees, which is often quite different from what an observer in either room would see. Transcription as analysis A transcript is not a neutral copy of a recording. It is a first analysis, and the conventions it uses determine what the analyst will notice. A transcript that records only words, in standard spelling, punctuated as written prose, makes consultations look far more orderly and fluent than they are, and hides almost everything that conversation analysis studies. The standard system for conversation analytic transcription was developed by Gail Jefferson over several decades and set out in a widely used glossary published in 2004. It aims to capture features of talk that participants demonstrably attend to: the timing of turns, including the length of pauses and the precise points at which one speaker starts while another is still talking; features of delivery such as stress, pitch movement, volume, stretching of sounds and cut-offs; audible breathing and laughter; and the speed of talk. Table 2 lists the conventions most commonly encountered in the medical literature. Table 2. Common transcription conventions in conversation analysis (after Jefferson). Symbol Meaning What it helps show (0.8) Silence timed in tenths of a second Delay before a dispreferred or difficult response (.) Micro-pause, under about two tenths of a second Hesitation within a turn [ ] Onset and end of overlapping talk Interruption, competition for the floor, or collaborative completion = Latching: no gap between turns or parts of a turn Rushing through a possible completion point to hold the floor : Stretching of the preceding sound Hesitation, emphasis or word search word- Cut-off Self-repair or abandonment of a turn °word° Noticeably quieter talk Delicate or reluctant content WORD Noticeably louder talk Emphasis or competition ↑ ↓ Marked rise or fall in pitch Surprise, contrast, or completion >word< Faster talk Hurrying, parenthetical remark .hh and hh Inbreath and outbreath Preparing to speak; sighs; laughter (( )) Transcriber's description Non-vocal conduct, such as a gaze shift Readers new to the field often find these transcripts forbidding, and it is fair to ask whether the detail is necessary. The answer is that the detail repeatedly turns out to matter to the participants. Silences of less than a second, for example, are routinely treated as meaningful. When a doctor proposes a treatment and the patient does not respond immediately, doctors frequently begin to modify or elaborate the proposal within a second or so, before the patient has said anything, treating the silence itself as incipient resistance. A transcript that did not time the silence would make the doctor's elaboration look unmotivated. Transcription is also slow. Estimates vary with the recording quality and the level of detail, but detailed transcription of one minute of talk commonly takes an hour or more. Most projects therefore produce a basic transcript of the whole corpus, then produce detailed transcripts only of the segments selected for close analysis. Automated speech recognition now produces serviceable first drafts of the words, but it does not time silences reliably, rarely marks overlap accurately, and tends to normalise disfluent speech into standard forms. It is useful as a starting point and dangerous as a finishing one. Alexa Hepburn and Galina Bolden's Transcribing for Social Research (2017) is the best practical guide to doing the work properly. For video, the transcript must also represent embodied conduct. Conventions for this are less standardised. The most widely adopted, developed by Lorenza Mondada, uses symbols to mark the onset and end of gestures, gaze shifts and body movements on a line beneath the talk, precisely aligned in time. Many medical studies use a simpler approach, noting significant embodied actions in double parentheses at the point where they occur. Transcribing languages other than English raises further issues. The usual practice is a three-line format: the original, a word-by-word gloss, and an idiomatic translation. Analysis should always be done on the original, with translation used only for presentation. Doing the analysis Conversation analysis does not begin with a hypothesis. It usually begins with what practitioners call unmotivated looking: repeated listening to and viewing of recordings, with transcripts, in search of something that seems orderly or puzzling. The analyst notices, for example, that patients who present a new symptom often add an account of why they have come now, or that doctors frequently say something while examining a patient that is not an instruction. The next step is to ask what the practice does. What problem does it solve for the speaker? How do recipients treat it? Once a candidate practice has been identified, the analyst builds a collection: every instance of the practice in the corpus, located by listening through the recordings or by searching transcripts. The collection is then examined systematically. What features do the instances share? In what sequential positions do they occur? How do recipients respond? What happens in the cases that look different? This is where the deviant-case analysis described in the previous chapter does its work. The analysis is refined until it accounts for the whole collection, including the exceptions, or until the analyst concludes that there are several distinct practices rather than one. Much of this work is collaborative. The "data session", in which a group of analysts watch or listen to a short fragment repeatedly and discuss what they see, is a characteristic institution of the field. It serves as a check on individual interpretation and as a way of training newcomers. Clinicians who attend data sessions often find them revelatory, because they see in slow motion interactional moves they make many times a day without noticing. Several analytic questions recur so often in medical research that they are worth listing. · Sequence. What action does this turn perform, what does it make relevant next, and what actually happens next? · Turn design. Why is this action done in this particular way, with these words, this grammar and this delivery, rather than some other way? · Overall structure. Where in the consultation does this occur, and how does its position shape its meaning? · Epistemics. Who is treated as knowing what, and who has the right to say it? · Deontics. Who is treated as entitled to decide what happens next? · Embodiment. How do gaze, posture, gesture and the handling of objects contribute to what is being done? These questions are not a checklist to be applied mechanically. They are the habits of attention that the method cultivates. Sampling and generalisation Qualitative studies of recorded consultations are often criticised for small samples, and the criticism deserves a careful answer. A conversation analytic study might be based on forty consultations, but it might analyse several hundred instances of a practice within them. The claim is not that forty consultations are statistically representative of all consultations, but that the practice has been described in enough detail, across enough variation, to be recognised and tested elsewhere. The claims are about how a practice works, not about how often it occurs in a population, and the appropriate test of such claims is whether they hold up in other data. That said, sampling decisions shape what can be found. A corpus from a single clinic may reflect local routines. A corpus limited to patients fluent in the majority language will say nothing about interpreted consultations. A corpus recorded only with consenting doctors may over-represent those confident in their communication. Good studies are explicit about these limits, and good syntheses draw on multiple corpora. The largest and most influential projects have been built with this in mind. The research on acute visits in American primary care that fed into Communication in Medical Care, for example, drew on several hundred recorded visits across multiple practices, allowing qualitative findings to be followed by quantitative tests. Corpora of this scale are expensive to build, and for that reason archived recordings with consent for secondary use are a valuable resource. The analysis of the patient's agenda by Naykky Singh Ospina and colleagues, published in 2019 and discussed in the next chapter, was a secondary analysis of 112 encounters originally recorded for trials of shared decision-making tools. From recording to finding A useful way to evaluate any study in this field is to ask three questions. Can I see the data on which the claim rests? Does the analysis attend to what participants did in response, rather than only to what the analyst thinks an utterance meant? And has the claim been tested against cases that might contradict it? Studies that satisfy all three are rare in any field. The best work in this one does, and it is that work on which the rest of this booklet draws. Chapter 3: Turn-Taking and the Architecture of the Visit The most basic fact about any conversation is that people take turns. Only one person usually speaks at a time, speakers change, and the transitions between them are managed with remarkable precision: gaps are typically a fraction of a second, overlaps are brief, and when either goes wrong, participants notice and fix it. In 1974 Harvey Sacks, Emanuel Schegloff and Gail Jefferson published a paper in the journal Language that described the machinery behind this, and it remains the most cited work in conversation analysis. Its title, "A simplest systematics for the organization of turn-taking for conversation", understates its ambition. It proposed that turns are built from units, such as sentences, clauses, phrases or single words, each of which brings the speaker to a point where transfer to another speaker becomes possible. At each such point, a set of ordered options applies: the current speaker may select the next, for instance by asking someone a question; failing that, another participant may self-select by starting to speak; failing that, the current speaker may continue. In ordinary conversation these options are open to everyone. Nobody is designated to ask the questions, nobody decides in advance how long each person may speak, and the order of topics is negotiated as the conversation goes along. Institutional interactions often modify this machinery, sometimes radically. In a courtroom, the order of speakers and the types of turn they may take are fixed by rule. In a news interview, interviewers ask and interviewees answer, and departures from this are noticeable and sanctionable. The medical consultation lies between these extremes. There is no formal rule that only the doctor asks questions, yet in practice doctors ask the overwhelming majority of them, and the distribution of speaking rights across the consultation is anything but equal. Asymmetry built into turns Understanding the turn-taking system clarifies where the doctor's control of the consultation comes from. The person who asks a question selects the next speaker and constrains what that speaker should do. A question also establishes a sequence that the questioner will ordinarily close, often by producing a third turn that acknowledges or assesses the answer before moving on. Mishler's cycle of question, response and assessment is, in these terms, a sequence in which one party holds both the first and the third positions. In each cycle the doctor opens the topic, the patient responds within the frame the question has set, and the doctor then decides whether to pursue it, close it, or move to another. Seen this way, the doctor's control does not require interruptions or displays of authority. It is produced simply by being the party who asks. Each question-answer pair returns the floor to the doctor, who then initiates the next. Patients who want to raise something else must either wait for a moment when the doctor explicitly invites it, or take the floor at a point where they have not been selected, which is interactionally costly. Candace West, in Routine Complications (1984), a study of family practice consultations, found that physicians asked the vast majority of the questions and that the relatively few questions patients asked were often marked by hesitation and self-correction, as if patients recognised that asking was not quite their business. This does not mean the patient's position is weak in every respect. The person who answers a question controls the content of the answer, and patients have considerable room to shape their responses. Tanya Stivers and John Heritage, in a 2001 study of comprehensive history-taking, showed that patients frequently answered "more than the question": responding to a yes-no question about, say, alcohol use with an elaboration that introduced information the doctor had not asked for, often about their circumstances, worries or explanations. These expansions allowed patients to bring their lifeworld into the consultation at points where the question format did not invite it. Doctors varied in how far they took these contributions up. The overall structure of the visit Beyond turns and sequences lies the level of overall structure: the recognisable phases through which a consultation moves. Byrne and Long's six phases were an early description. John Heritage and Jeffrey Robinson, drawing on recordings of acute primary care visits in the United States, proposed a similar sequence: opening, problem presentation, information gathering through history-taking and physical examination, diagnosis, treatment, and closing. Robinson showed in a 2003 paper that participants orient to this structure as a normative sequence, which is to say that they treat movement from one phase to the next as expected and treat departures as noticeable. Patients who ask about treatment before the doctor has examined them, for example, may be told that the doctor will "get to that", which shows that both parties recognise an order in which things should happen. The overall structure matters because each phase has its own local rules of participation, and the patient's opportunities to shape the consultation vary across them. In the problem presentation, patients are expected to speak at length. In history-taking, they are expected to answer questions. During the physical examination they are largely silent. When the diagnosis is delivered, they are positioned as recipients, though, as Chapter 6 shows, they may have more room than this suggests. Knowing where in the structure an utterance falls is therefore essential to understanding what it does and what the patient could have done instead. Table 3 summarises the phases of the acute primary care visit as the research has described them. Table 3. Phases of the acute primary care visit and what analysts attend to in each. Phase Usual initiator Patient's normative role Analytic focus Opening Doctor Greets; waits to be invited to speak Form of the opening question; timing to gaze and records Problem presentation Patient, when invited Extended telling of the concern Length before interruption; how the visit is justified History-taking Doctor Answers questions Question design; answers that expand beyond the question Physical examination Doctor Complies; largely silent Online commentary; embodied coordination Diagnosis Doctor Receives; may respond or resist Evidential basis displayed; patient uptake Treatment Doctor, sometimes patient Accepts, questions or resists Recommendation format; patient participation Closing Doctor Raises further concerns if invited Form of final solicitation; late-arising concerns The opening seconds The problem presentation phase has been studied more closely than any other, because it is the patient's main structural opportunity to set the agenda of the consultation in their own terms. It is also where one of the field's most famous findings originated. In 1984 Howard Beckman and Richard Frankel published a study in the Annals of Internal Medicine based on audio recordings of seventy-four office visits. They found that physicians frequently redirected patients before they had completed their opening statement of concerns, typically by asking a closed question about the first concern mentioned. On average, the redirection came after eighteen seconds. Only a minority of patients completed their opening statement, and once interrupted, patients seldom returned to it. The implication was that doctors were frequently beginning to investigate the first problem mentioned before learning whether it was the most important, or even the only one. In 1999 Kim Marvel and colleagues revisited the question with recordings of 264 visits to family physicians and found that the picture had changed little. Physicians solicited the patient's concerns in most visits, but patients were typically redirected after about twenty-three seconds, and relatively few completed their initial statement. Late-arising concerns, those first mentioned near the end of the consultation, were more common when the agenda had not been fully elicited at the start. In 2019 Naykky Singh Ospina and colleagues analysed 112 recorded encounters from trials of shared decision-making tools in American primary and specialty care. Clinicians elicited the patient's agenda in only 36 percent of encounters, less often in specialty care than in primary care, and when they did so, patients who were interrupted were interrupted after a median of eleven seconds. Patients who were not interrupted took a median of six seconds to state their concern. That last figure is often overlooked. The fear that motivates early interruption is that patients, left uninterrupted, will talk for many minutes. The evidence suggests otherwise. Most patients, if allowed, finish their opening account quickly. The cost of letting them do so is small; the cost of not doing so may be that the consultation addresses the wrong problem. How openings are designed Interruptions are only part of the story. How the doctor invites the patient to speak shapes what the patient says. Heritage and Robinson, in studies published in 2006, distinguished between opening questions that invite the patient to present a new concern, such as "What can I do for you today?", and those that invite confirmation of something the doctor already knows or assumes, such as "I understand you're having some headaches?" or "Sore throat, huh?". Both are common. The first kind, which they called general inquiries, gives the patient the floor for an extended telling in their own terms. The second, which requests confirmation, invites a short response and allows the doctor to proceed quickly to history-taking. Patients responded differently to the two. General inquiries elicited longer problem presentations that contained more discrete symptoms. Robinson and Heritage also reported that patients' satisfaction with how well the physician listened was higher when visits opened with a general inquiry. None of this means that confirmatory openings are wrong. They can be efficient when a nurse or a form has already recorded the complaint, and patients sometimes appreciate that the doctor has read about them. But they carry a cost that is invisible from the doctor's side: they cast the recorded complaint as the reason for the visit, and patients may not correct that framing if it is incomplete. A different kind of opening is common in follow-up visits: "How are you doing?" or "How's the knee?". These invite the patient to report on a known problem and make it harder to introduce a new one. Robinson showed that patients distinguish carefully between opening questions that treat the visit as a new-problem visit and those that treat it as a follow-up, and shape their responses accordingly. A patient with a new concern at a follow-up visit must find a way to raise it that departs from the frame the doctor has set. Justifying the visit Heritage and Robinson also drew attention to something patients do that doctors seldom notice: they justify their decision to come. Patients' problem presentations are often built to show that the concern is "doctorable", that is, worthy of medical attention and not trivial. A patient might describe how long the symptom has persisted, what remedies they have already tried, or which family member urged them to come. Timothy Halkowski showed that patients often narrate how they came to recognise a symptom as a problem, presenting themselves as reasonable people who did not rush to the doctor at the first twinge. These practices reveal a pressure on patients that shapes their contributions throughout the consultation: the worry of being judged to have wasted the doctor's time. This matters for diagnosis. A patient working to establish the legitimacy of a visit may emphasise the features of a symptom that make it seem serious enough to justify attendance, or may downplay features that seem embarrassing or trivial. What reaches the doctor is not a neutral inventory of symptoms but a report shaped by its speaker's concern to appear reasonable. The screen as participant Turn-taking in the modern consultation is complicated by the computer. When a doctor types during a patient's account, the patient often slows down, pauses or restarts, as if timing their contributions to the doctor's attention. When the doctor turns to the screen to look something up, a silence may open that neither party is quite sure how to fill. Some doctors narrate what they are doing ("I'm just checking your last blood results"), which keeps the patient oriented; others work in silence, which can leave the patient uncertain whether to continue speaking. Studies of consultations with electronic records have found that doctors spend a substantial proportion of the visit looking at the screen, and that the screen's placement affects how easily attention can be shared. A screen positioned so that both parties can see it allows doctors to show results and invite the patient into the process. A screen facing only the doctor creates a divided consultation in which the patient must compete for attention with the record. Remote consultations change the turn-taking system further. On the telephone, gaze and posture are unavailable, and silences become harder to interpret: is the doctor thinking, typing, or waiting? On video, there is often a slight transmission delay that disrupts the fine timing of turn transitions, producing more overlapping starts and more awkward gaps. Participants compensate with more explicit verbal management of the floor, but the subtle cues that let patients judge when they may speak are attenuated. Research on these formats is still developing, and it is one of the places where the methods of this booklet are most needed. Why the architecture matters It is tempting to treat all this as etiquette, the conversational equivalent of a good bedside manner. But the architecture of the consultation determines what information reaches the clinician. A patient who is interrupted after eleven seconds, whose visit opened with a question that confirmed only the recorded complaint, and who is working hard to seem reasonable, is in a structurally poor position to disclose the symptom that worries them most. No one needs to have behaved badly for this to happen. The system of turns and phases produces it by default. That is the argument of this booklet in its simplest form, and the next chapter examines the doctor's main instrument within that system: the question. Hashtags: #DiscourseAnalysisInHealthcare #DoctorPatientInteraction #MedicalCommunication #ConversationAnalysis #CriticalDiscourseAnalysis #InteractionalSociolinguistics #RoterInteractionAnalysisSystem #ClinicalConsultations #TurnTaking #ConsultationArchitecture #QuestionDesign #PatientAgenda #MedicalPower #InstitutionalDiscourse #DiagnosticCommunication #Epistemics #Deontics #JeffersonTranscription #EmbodiedInteraction #MultilingualConsultations #SharedDecisionMaking #TreatmentNegotiation #CommunicationGaps #DiagnosticTalk #FutureOfHealthcareDiscourse

Download the Book (PDF): Introduction A longitudinal study is the only instrument social and biomedical science has for watching a life unfold. Everything else — the cross-sectional survey, the case-control study, the randomised trial with a twelve-month endpoint — takes a slice and reasons about the rest. A cohort followed for forty years does not have to reason about it. It watches. That is the promise, and it is why the great cohorts have a standing in their fields that almost nothing else achieves. The Framingham Heart Study, begun in 1948 with 5,209 residents of a Massachusetts town, is the reason the phrase "risk factor" exists. The British birth cohorts — 1946, 1958, 1970, 2000 — have given four generations of British social policy its evidentiary spine. The Dunedin Multidisciplinary Health and Development Study, tracking 1,037 people born in one New Zealand city in 1972 and 1973, has produced findings about childhood self-control, adolescent cannabis use and the pace of biological ageing that no other design could have reached. These studies are slow, expensive, administratively awkward, and worth every difficulty. But the promise carries a condition that is easy to state and brutally hard to meet: the people have to stay. A cohort study's scientific value is not established at baseline by a good sampling frame. It is produced continuously, wave after wave, by the study's ability to keep finding its participants, keep persuading them to answer, and keep measuring them in a way that does not itself change what is being measured. A study that begins with a beautiful probability sample of 10,000 and arrives at wave twelve with 3,400 people who answer the phone has not preserved its sample. It has acquired a new one, selected by a process nobody designed and nobody fully understands. This is the problem the booklet addresses, and the argument it makes is a specific one: in a long-running panel, retention, measurement and analysis are not three separate departments but one system, and the design choices that keep people participating are the same choices that determine whether the statistical adjustments for those who leave can be believed. That claim has practical teeth. It means that a decision made in the field office — whether to shorten the questionnaire, whether to accept a web response in place of an interview, whether to pay £30 or £50, whether to send a birthday card — is a decision about the credibility of the missing-data model that will be fitted twenty years later. It means that the auxiliary variables an analyst wishes were available to make a missing-at-random assumption plausible have to have been collected, deliberately, long before anyone knew they would be needed. It means that panel conditioning — the way repeated measurement alters the thing measured — is not a nuisance to be acknowledged in a limitations paragraph but a design parameter that can be traded against retention. And it means that the standard defensive move of longitudinal analysis, the multiple imputation model with forty auxiliary variables and a hundred imputations, is only as good as the fieldwork that generated its inputs. Imputation cannot manufacture information. It can only propagate, honestly, the information a study took the trouble to collect. The three topics named in the title are conventionally taught apart. Retention belongs to survey methodology and field operations. Missing data belongs to statistics. Panel conditioning belongs to a small measurement literature that most cohort investigators encounter once and then forget. Treating them separately is how studies end up with sophisticated imputation applied to data whose missingness is driven by mechanisms the imputation model has no variables for, and with engagement strategies that quietly reshape the construct they were meant to preserve. There is a further reason to write about this now. The operational environment of cohort studies has changed more in the last fifteen years than in the previous fifty. Household telephone coverage, the backbone of panel tracing since the 1970s, has collapsed. Response rates to unsolicited contact of any kind have fallen across every developed country, in some cases by more than half within a generation. Postal address registers have become less stable as housing has become less stable. At the same time, a set of new instruments has arrived: participant portals, smartphone apps, wearable accelerometers and continuous glucose monitors, linkage to administrative and electronic health records, passive location and app-use data, and the possibility of measuring some things without asking anyone anything. These developments cut both ways, and the field has not fully reckoned with either edge. Remote and passive measurement genuinely reduces burden and can lift retention among people who would never sit through a three-hour clinic visit. It also introduces differential technology access, device attrition that is itself informative, mode effects that break comparability with thirty years of prior waves, and a kind of engagement — the notification, the streak, the dashboard — borrowed from consumer products designed to maximise something other than data quality. A study that adopts app-based follow-up has not solved its retention problem. It has exchanged a well-understood one for a poorly characterised one. The same is true of algorithmic imputation. Machine-learning methods for filling in missing values are now widely available and, on the benchmarks usually reported, impressively accurate. But predictive accuracy on an artificially masked complete dataset is not the criterion that matters for inference. The criterion is whether the imputation procedure is congenial to the analysis model, whether it propagates uncertainty correctly, and whether it operates under an assumption about the missingness mechanism that the study's design has made defensible. A random forest that predicts held-out values well can produce confidence intervals that are badly wrong, and it will do so silently. The field has a real appetite for methods that seem to make missingness go away. Nothing makes missingness go away. So the booklet is organised around the life of a panel rather than the divisions of a literature. It begins with what a cohort is actually for, because the estimand determines what kind of loss matters — a study of within-person change tolerates a different pattern of attrition than a study of population prevalence, and studies routinely confuse the two. It then works through the arithmetic of loss and its anatomy; the field infrastructure of tracing and contact; the digital layer and what it does and does not deliver; the economics and ethics of burden, incentives and consent across decades; and panel conditioning, the least-managed of the major threats. The second half turns to analysis. Two chapters deal with missing data: first the assumptions, because the choice between mechanisms is a substantive claim about the world and not a statistical option, and then the methods, because the methods only mean what their assumptions license. A chapter on survivorship and selection deals with the specific pathologies that afflict ageing cohorts, where death is not missingness but a competing event, and where conditioning on survival can produce associations that reverse the truth. The last chapter is about institutional memory: the documentation, paradata and governance that let a study outlive the careers of the people who designed it, which for a multi-decade cohort is not an administrative footnote but a precondition of the science. A word about the intended reader. This is written for people who run, analyse, fund or review cohort studies, and for the wider population of researchers who use cohort data without having been in the field office. It assumes comfort with regression and with the idea of a statistical model, but it does not assume familiarity with pattern-mixture parameterisations or inverse probability of censoring weights, and it explains those where they arise. It is unapologetically practical. Where there is a real disagreement in the methodological literature, it says so and takes a position. What it will not do is pretend that these problems have solutions. They have management strategies. A cohort study that runs for forty years will lose people, and some of that loss will be informative in ways no model can fully repair. The honest posture is not to claim that adjustment has restored the original sample — it has not — but to make the assumptions explicit, to design the study so that those assumptions are as weak as possible, and to report what happens to the conclusions when they fail. That posture is less satisfying than the alternative and considerably more scientific. One last framing point. The dominant metaphor for attrition is leakage: the study is a vessel, participants drain out, retention is plumbing. It is a bad metaphor, because it implies that the people who remain are unchanged by remaining. They are not. They have been interviewed nine times, they have learned what the study wants, they have formed a relationship with an institution, and in some measurable respects they have become different people from the ones who left — partly through selection, partly through the act of being studied. A better metaphor is cultivation: the study and its participants constitute a long relationship, which shapes both parties, and which has to be maintained on terms that neither exhausts the participant nor contaminates the measurement. Everything that follows is an elaboration of that relationship and of what it costs. Chapter 1: What a Cohort Is For Before anything can be said about who leaves a study and what to do about it, there has to be an answer to a prior question: what quantity is the study trying to estimate? The answer sounds obvious until you try to write it down, and the act of writing it down changes what counts as damaging attrition. Two studies can lose the same forty per cent of their participants and be in entirely different positions — one crippled, the other barely affected — because they were aiming at different targets. This chapter is about specifying the target. It is the least glamorous chapter in the book and the one with the highest return, because almost every argument about missing data that goes in circles is an argument in which the disputants are estimating different things. Three families of target quantity Longitudinal designs are recruited into service for three broad purposes, and the distinction matters enormously for attrition. The first is descriptive population inference over time: what proportion of British 46-year-olds have hypertension; how median household wealth in Germany evolved between 1995 and 2020; what share of a cohort completed tertiary education. Here the sample is meant to represent a population at each point, and the estimand is a population-level quantity at a specified time. Attrition is directly threatening, because any differential loss immediately biases the population estimate. If people with lower education leave faster, the estimate of educational attainment drifts upward wave by wave for reasons that have nothing to do with anyone's education changing. Descriptive panel estimates are the most fragile thing a panel produces, and they are also what journalists and policymakers most want from it. The second is within-person change: how does cognitive function decline between 60 and 80; does an individual's blood pressure trajectory bend after a job loss; how much does life satisfaction rebound after divorce. Here each person serves as their own control, and much of the confounding that plagues cross-sectional work is differenced away. Attrition is less immediately corrosive: if the people who remain have the same rate of change as those who left, even a heavily selected remaining sample can estimate the change parameter without bias. That is a real assumption, not a free pass, but it is a substantially weaker one than the assumption needed for descriptive estimates. It is also why fixed-effects and mixed-model analyses often survive attrition that would destroy a prevalence estimate from the same data. The third is causal effect estimation: does redundancy cause depression; does adolescent cannabis use reduce adult IQ; does childhood adversity accelerate biological ageing. Here the estimand is a contrast between potential outcomes in a defined target population, and attrition threatens in a more subtle way. Loss to follow-up is a form of censoring, and it biases the effect estimate to the extent that it depends jointly on the exposure and the outcome. Loss that depends only on exposure, or only on measured covariates, is manageable; loss that depends on the outcome given exposure is the dangerous case. Framing attrition as censoring rather than as "missing data" is more than terminological — it brings the whole apparatus of causal inference under censoring, including inverse probability of censoring weights and g-methods, to bear on a problem that survey methodology often treats as a weighting exercise. Most large cohorts serve all three purposes simultaneously, which is precisely the trouble. A study designed as a causal machine gets used to produce national prevalence estimates, and the attrition adjustment that was adequate for the first is quietly assumed adequate for the second. The single most useful discipline a study team can adopt is to state, for each headline output, which family it belongs to, and to accept that different adjustment strategies and different weights apply. Closed cohorts, open panels, and what "the sample" means at wave twelve The structural design of the study determines what replacement is possible and what it means. A closed cohort recruits once and follows those people. Birth cohorts are the pure case: everyone born in Britain in one week of March 1958, followed ever since. Nobody new can join, because nobody else was born that week. The Framingham original cohort, the Dunedin study, the Whitehall II civil servants recruited in 1985–88 — all closed. In a closed cohort, attrition is irreversible. Every person lost is lost permanently, sample size decays monotonically, and the only defences are retention and analytical adjustment. An open or rotating panel allows replenishment. Household panels like the Panel Study of Income Dynamics (running since 1968), the German Socio-Economic Panel (since 1984) and the UK's Understanding Society (since 2009, succeeding the British Household Panel Survey) follow households and their descendants, adding new entrants through births, household formation and, periodically, refreshment samples drawn afresh. The Health and Retirement Study adds a new birth cohort every six years so that it can continue to represent Americans over 50 rather than an ever-older group of survivors. Rotating panels — common in labour force surveys — cycle households in and out on a fixed schedule, capping the conditioning any one household experiences. Replenishment is not a cure for attrition. It restores sample size and it restores cross-sectional representativeness, both genuinely valuable. It does nothing for the longitudinal estimand: a person who joined in 2019 cannot contribute to an analysis of change since 1998. Studies that refresh can end up with a comfortable-looking N and a long-run analytic sample that is small and heavily selected. When reading a panel paper, the question is not how many people are in the study but how many contributed data at both ends of the interval being analysed. There is a third structure worth naming, increasingly common: the linkage-anchored cohort, in which active follow-up is supplemented or partly replaced by routine record linkage — mortality registers, cancer registries, hospital episode data, tax and benefit records, education records. UK Biobank, recruited between 2006 and 2010 with about 500,000 participants, was designed on this model from the start: a substantial baseline assessment followed by long-term passive follow-up through health records, with active re-contact for subsets. Linkage changes the attrition problem profoundly, and mostly for the better, because the outcome ascertainment does not depend on the participant answering anything. What it does not do is provide the repeated self-reported measures — mood, relationships, attitudes, behaviours — that most longitudinal social science needs. A cohort can be nearly complete for mortality and 55 per cent complete for depressive symptoms at the same wave. The horizon problem Multi-decade studies face a design difficulty that shorter ones do not: the questions the study will be asked to answer have not been invented yet. This is not hyperbole. The 1958 National Child Development Study was designed to investigate perinatal mortality. It has since been used to study social mobility, obesity trajectories, adult literacy, mental health, and epigenetic ageing — none of which were on anyone's mind in 1958. The Dunedin study's findings on the gene–environment interplay in depression, and on the developmental course of self-control, rest on measures collected long before the relevant hypotheses existed. The 1946 birth cohort's value for dementia research comes from childhood cognitive tests administered for entirely different reasons. The horizon problem has two consequences for the subject of this booklet. First, it means that measurement decisions are irreversible in a way that analytic decisions are not. An analyst in 2045 can fit any model they like to the data. They cannot go back and ask a question that was not asked in 2005. This asymmetry argues for collecting more than the current protocol requires — particularly variables that predict both future attrition and likely future outcomes, because those are exactly the auxiliary variables that will make a missing-at-random assumption defensible. Contact history, residential stability, health service use, cognitive function, conscientiousness, and interviewer assessments of respondent engagement are cheap to collect and disproportionately useful later. A study that skimps on them in 2010 to save four minutes of interview time has made an expensive trade it will not notice for fifteen years. Second, it means the study's retention strategy has to be robust to changes it cannot foresee. Telephone-based tracing was an excellent strategy in 1990 and is a poor one now. Any strategy that depends on a single channel — a single technology, a single funding source, a single charismatic principal investigator — is fragile over forty years. The cohorts that have survived longest are not the ones that picked the best contemporary technique; they are the ones that maintained multiple redundant links to their participants and a stable institutional home. Estimands and the design of attrition tolerance Putting these pieces together yields a practical procedure that too few studies carry out explicitly. Start by listing the study's priority estimands — perhaps five to ten, spanning the three families. For each, ask what pattern of loss would bias it. For a descriptive prevalence estimate, any loss correlated with the outcome is a problem. For a within-person slope, only loss correlated with the slope is a problem, which is a different and often smaller set of people. For a causal contrast, loss correlated with the outcome conditional on exposure and covariates is the problem. Then ask: what would we need to have measured to adjust for that pattern? That question generates a list of auxiliary variables, and that list should drive questionnaire content, not the other way round. The most common failure in longitudinal design is that the auxiliary variable list is assembled by a statistician after five waves of data have already been collected without it. Finally, ask what level of loss the study can tolerate before each estimand becomes unreportable, and write that down in advance. Studies almost never do this, and as a result they slide into a position where the wave-fourteen prevalence estimate is published with a caveat that nobody reads, on a sample retaining 41 per cent of baseline, where a pre-committed rule would have said: report change, not levels. Table 1 sets out how the three estimand families differ in what they require and what threatens them. Table 1. Estimand families and their vulnerability to attrition. Estimand family Typical question Primary threat from loss Key defence Descriptive population Prevalence, means, distributions at a time point Any loss related to the outcome Weighting to external population benchmarks; refreshment samples Within-person change Trajectories, rates of decline, response to events Loss related to the rate of change, not the level Mixed models with all available waves; auxiliary predictors of slope Causal contrast Effect of an exposure on a later outcome Loss related jointly to exposure and outcome Inverse probability of censoring weights; g-methods; sensitivity analysis Note: the three families are not mutually exclusive, and most cohorts produce outputs of all three kinds from the same data. What this reframing buys The payoff of estimand-first thinking is that it converts vague anxiety into specific, checkable questions. "We have 52 per cent retention, is our study still valid?" is unanswerable. "We have 52 per cent retention; our headline output is the within-person association between mid-life obesity and cognitive decline; is loss to follow-up plausibly related to the rate of cognitive decline conditional on baseline cognition, education, and health?" is a question that can be interrogated with data — by examining whether early cognitive slopes predict later dropout, by comparing those who missed one wave and returned against those who left permanently, and by running the analysis under an assumed departure from ignorability. That is the shape of every honest attrition analysis in this book: not a claim that the sample is still representative, but a demonstration that the specific quantity being estimated is robust, or a clear statement of how far it would have to be wrong for the conclusion to change. The rest of the booklet works through the mechanisms that generate the loss, the operational practice that limits it, and the analytical tools that make what remains usable. But none of that machinery is interpretable without first knowing what the study is for. A panel that cannot say what it is estimating cannot say whether it has been damaged, and studies in that position tend, understandably, to assume they have not been. A worked case: the same data, two verdicts It helps to see how sharply the estimand determines the verdict. Consider a hypothetical but entirely typical situation. A cohort recruited 8,000 adults aged 45 in 2000 and has followed them at five-yearly intervals. At the 2025 wave, aged 70, 4,100 people provided data: 900 have died, 1,400 refused or were untraceable, 1,600 were alive and enrolled but missed this particular wave. Baseline education, occupational class and self-rated health all predict who is still responding, with the expected gradient — graduates are about fifteen percentage points more likely to be in the responding group than those who left school at sixteen. Now ask two questions of these data. Question one: what proportion of 70-year-olds in this population have clinically significant depressive symptoms? This is a descriptive population estimate, and it is in serious trouble. Depressive symptoms are strongly patterned by education and occupational class in the same direction as the response gradient. The responding sample is better educated than the surviving population, so the raw estimate will be too low. Weighting on baseline education, class and health will help, and may help a lot, but it can only correct for what is in the weighting model. If refusal at wave six is driven in part by current depressed mood — which is plausible, since depression reduces the energy available for a two-hour interview — the weights cannot touch that, because current mood is exactly what is missing. The honest report here is an estimate with a wide bound, plus a sensitivity analysis showing how far the prevalence would move if non-respondents had, say, 1.5 times the odds of depression conditional on the weighting variables. Question two: among people who experienced bereavement between 2015 and 2020, how much did depressive symptoms rise relative to their own prior trajectory? This is a within-person change estimand, and its position is far stronger. The educational gradient in response does not bias it unless graduates' response to bereavement differs from non-graduates', which is a separate and empirically checkable proposition. More importantly, a mixed model uses everyone who contributed at least two observations, including the 1,600 intermittent responders whose earlier waves are complete. The analytic sample for this question is not 4,100 but closer to 6,500, and its selection is on characteristics that the model conditions on directly. Same dataset, same attrition, two entirely different levels of concern. A study that reports both outputs in the same paper with a single sentence about response rates has misled its readers about at least one of them. The example also demonstrates why the categories in the loss have to be separated rather than pooled into "attrition". Of the 3,900 people not observed at wave six, 900 are dead. For the depression prevalence question, the dead are not missing — they are simply not members of the target population, which is living 70-year-olds. For a question about lifetime cumulative risk of depression, they are not missing either, but their absence changes the estimand in a way that needs to be stated. For a question about cognitive decline, the dead are the single most important group in the analysis, because decline and death are jointly determined and analysing only survivors is a known route to nonsense. Chapter 9 deals with this at length. The point here is that mortality, refusal and temporary non-contact are three different phenomena that happen to produce the same blank cell in a data matrix, and no adjustment method can distinguish them if the study's own records do not. Writing the estimand down The practical recommendation from this chapter is unfashionably bureaucratic: maintain a living document that lists the study's priority estimands, and revisit it at every funding renewal. For each estimand, record the target population (including whether the dead are in it), the time point or interval, the quantity, the pattern of loss that would bias it, the auxiliary variables collected to adjust for that pattern, and the pre-committed threshold below which the study will stop reporting it unadjusted. Half a page each. A study with twelve priority estimands has a six-page document that determines most of its questionnaire content, most of its retention priorities, and all of its analytic defaults. The reason this is worth the effort has nothing to do with administrative tidiness. It is that arguments about attrition, once a study is well into its life, are almost always conducted in the abstract — is 48 per cent retention acceptable? — and abstract arguments about attrition have no resolution. Grounding them in a specific list of quantities turns an unresolvable dispute into a set of tractable technical questions, each of which has an answer, and several of which turn out to be reassuring. Chapter 2: The Arithmetic of Loss Attrition is usually reported as a single number — "retention was 68 per cent at wave eight" — and that number conceals almost everything worth knowing. It conceals the shape of the loss over time, the distinction between people who are gone and people who are merely absent, the difference between a household that moved and a household that refused, and the fact that mortality and refusal have opposite implications for most analyses. This chapter builds the vocabulary and the arithmetic needed to say something useful. Compounding, and why wave-on-wave rates deceive The first thing to understand about panel loss is that it multiplies. A study that retains 90 per cent of its responding sample at each wave — a rate any field director would be pleased with — retains 0.9^10, or 35 per cent, after ten waves. At 95 per cent per wave, ten waves leaves 60 per cent. At 85 per cent, it leaves 20 per cent. This is elementary, and it is routinely mis-experienced by study teams, because the number they see each round is the wave-on-wave rate, which always looks acceptable. The cumulative figure only becomes visible when someone plots it, and by then several waves of decisions have been made on the basis of the comfortable number. A useful discipline is to report cumulative retention against the original baseline as the headline operational metric, with wave-on-wave as a secondary diagnostic. It changes the felt urgency of a two-point drop. The compounding is not, in practice, geometric. Real panels show a characteristic profile: a large drop between the first and second wave, a smaller drop between the second and third, and then a long shallow decline that is much flatter than the early losses would predict. The reason is selection on propensity to participate. The first wave loses the people who were marginal about joining at all; those who come back a second time are, by revealed preference, more committed, and their subsequent per-wave attrition is lower. By wave five the remaining sample is dominated by habitual participants whose annual attrition may be only two or three per cent. This is good news operationally — the study is not on an exponential path to zero — and bad news analytically, because it is a precise description of increasing selection. The stable core is stable because it is different. The practical implication is that the early waves are where retention investment pays the largest dividend. A pound spent on converting a wave-two refusal buys, in expectation, many more person-waves of future data than a pound spent at wave nine, both because the person has more waves ahead and because early converts often become long-term participants. Studies that ramp up their retention effort as the panel ages, in response to the visible decline, have the allocation backwards. An anatomy of not-responding "Missing" is a data structure. It is not a phenomenon. The phenomena behind a blank cell are various, and the first job of any serious attrition analysis is to separate them. Unit non-response at baseline is the failure to recruit in the first place. It is frequently ignored in discussions of attrition because it happens before the study exists, but it determines the starting point. A cohort that achieved a 42 per cent baseline participation rate — not unusual for a modern volunteer-recruited study — begins already selected, and all subsequent retention, however excellent, operates on a non-representative base. UK Biobank is the canonical example and has been open about it: its participants are healthier, wealthier and less likely to smoke than the general population of the same age. The consequence, well described in the literature on this cohort, is that prevalence estimates from it are not generalisable, while exposure–outcome associations within it are often much less affected. This is the estimand distinction from Chapter 1 reappearing at baseline. Wave non-response is a person alive and enrolled who does not provide data at a given wave. This is the workhorse category and it subdivides further: · Non-contact: the study could not reach them. Wrong address, disconnected number, unanswered door. Non-contact is largely an operational failure and is the most tractable category — better tracing solves much of it. · Refusal: the study reached them and they declined. Refusals are informative in a way non-contacts are not, because they reflect a decision that may be related to current circumstances — illness, distress, a bad experience at the last interview. · Incapacity: too ill, too cognitively impaired, hospitalised, in institutional care. This category grows dramatically in ageing cohorts and is intensely informative, because incapacity is usually related to exactly the outcomes under study. A cognition study that loses people to dementia-related non-response and treats those absences as ignorable is making a serious error. · Other unavailability: abroad, in prison, homeless, in a domestic situation that makes participation unsafe. Item non-response is a completed interview with specific questions unanswered. It is far more common than teams assume and its causes are specific — income questions, sexual behaviour, substance use, questions about a deceased spouse. Item missingness is often the easiest to impute well, because the respondent has supplied a rich body of contemporaneous data to condition on. It is also the one place where "missing not at random" is most obviously true: people who refuse to state their income are not a random subset of earners. Attrition proper, in the strict sense, means permanent withdrawal — formal consent withdrawal, a firm final refusal, emigration with no trace, or death. Only death is unambiguously permanent, and even a formal withdrawal request may cover future contact while permitting continued use of data already collected, depending on the consent architecture. Death deserves separate treatment throughout, and gets a chapter. For now: death is not missing data. The person has not failed to report; there is nothing to report. Treating death as missingness and imputing post-mortem blood pressure is an error that sounds absurd stated baldly and happens routinely in practice whenever a default imputation routine is pointed at a cohort file without filtering on vital status. Table 2 sets out these categories with what each implies for analysis. Table 2. Forms of non-participation in a panel study and their analytic implications. Category Typical drivers Reversible? Usual missingness assumption Main defence Baseline non-response Volunteer selection, access, trust No Often MNAR on health and SES External benchmark weighting; restrict to internal comparisons Non-contact Mobility, obsolete contact details Yes Often close to MAR given mobility predictors Tracing infrastructure; multiple contact channels Refusal Burden, prior bad experience, current distress Often MAR at best; MNAR plausible Refusal conversion; incentives; mode choice Incapacity Illness, cognitive decline, institutionalisation Rarely MNAR by construction for health outcomes Proxy respondents; short forms; linkage Item non-response Sensitivity, recall difficulty, question design Wave-specific MAR given rich concurrent data Multiple imputation; better question wording Death Mortality No Not missingness; competing event Joint models; survivor-specific estimands Monotone and non-monotone patterns The shape of the missingness matrix matters for both interpretation and method. Monotone missingness means that once a person is absent, they never return: the data form a staircase. This is the classic dropout pattern, and it is analytically convenient, because it permits sequential modelling — the probability of being observed at wave t can be factorised into a chain of conditional probabilities given observation at t−1, which is exactly what inverse probability of censoring weighting exploits. Non-monotone or intermittent missingness means people skip waves and come back. This is the empirical reality of almost every real panel: a substantial share of any wave's non-respondents will respond again later. In household panels the figure is often a quarter to a third of the wave's non-respondents returning at the next contact. Intermittent responders are an enormously valuable group and are frequently mishandled. They are valuable because they let the study observe the characteristics of non-respondents directly — someone who was absent at wave five and present at wave six can be asked about wave five, and their wave-six data tells you what wave-five non-respondents look like, at least for the returning subset. This is the closest thing a panel has to a natural experiment on its own missingness, and it is the empirical basis for most defensible MAR arguments. They are mishandled in two ways. First, many analyses restrict to "complete cases" or to people present at both ends of an interval, discarding intermittent responders whose data is perfectly usable in a likelihood-based model. Second, field operations sometimes drop people from the issued sample after two consecutive non-responses, a rule that permanently converts a recoverable intermittent responder into an attrition case. The rule exists because re-issuing costs money and yield falls with each consecutive miss, which is true. But it is a choice that manufactures monotone missingness, and it should be made deliberately with knowledge of what it destroys, not adopted as a default because a fieldwork contractor proposed it. Understanding Society and several other major panels have moved towards keeping non-respondents in the issued sample for longer, sometimes with a lighter-touch approach — a short web questionnaire instead of a full interview — precisely to preserve the possibility of return. The evidence from such efforts is that a meaningful fraction do come back, and that they differ systematically from continuous responders in ways that matter. Who leaves: the empirical regularities Across an enormous range of studies, in different countries and on different topics, the same predictors of attrition recur. They are worth knowing because they constitute the prior that any study's own attrition analysis should be checked against. Residential mobility is the strongest and most consistent single predictor, and in most panels it operates mainly through non-contact rather than refusal. Young adults, renters, recent migrants and people in unstable housing move more and are lost more. The effect is so dominant in young-adult waves that many studies see their worst retention between ages 18 and 30 and then a recovery as the cohort settles. Socioeconomic position predicts attrition robustly: lower education, lower income, manual occupation and unemployment are all associated with higher loss. The mechanisms are mixed — mobility, less flexible time, lower perceived relevance of the study, and in some settings greater distrust of institutions. Minority ethnic status and migration background are associated with higher attrition in most Western panels, driven by language, mobility, international migration and, in some settings, by mistrust arising from historical mistreatment of minority communities by research and medical institutions. This is not a fixed feature of those populations but a feature of how studies have engaged with them; studies that invest in community relationships, bilingual fieldwork and culturally appropriate materials narrow the gap considerably. Poor health and health decline predict attrition, and this is the most analytically dangerous regularity, because health is usually the outcome. In ageing cohorts the effect is strong: people who become seriously ill, cognitively impaired or institutionalised stop participating. The resulting sample is systematically healthier than the surviving population, a phenomenon distinct from survivorship bias and additive to it. Male sex is associated with modestly higher attrition in most general-population panels. Prior response behaviour is by far the best available predictor of future response, and it is measured directly from the study's own records. Number of prior waves completed, whether the last wave was completed, the number of contact attempts required, whether an incentive was needed, interviewer-rated cooperativeness, item non-response rate at the last interview and interview duration all predict subsequent participation, often better than any substantive characteristic. This matters greatly for adjustment: paradata are the cheapest and most powerful auxiliary variables a study has, and studies that do not retain them have thrown away their best tool. Single-person households and recent household dissolution predict loss in household panels, because the study's link to a household breaks when the household does. A caution about these regularities: they describe who leaves, which is only half of what adjustment needs. The relevant question for bias is whether, conditional on everything measured, the people who left differ from those who stayed on the outcome. A predictor of attrition that is fully measured and included in the adjustment model is, in a sense, harmless. It is the unmeasured component of the attrition process that does the damage. This is why a long list of significant predictors of dropout is not, by itself, evidence that the study is biased — and equally why a finding that "respondents and non-respondents did not differ significantly on baseline characteristics" is far weaker reassurance than it is usually presented as being. Non-significance on a handful of baseline variables in a study with limited power to detect differences tells you very little about the unmeasured mechanism. What a good attrition report looks like Most published cohort papers dispose of attrition in two sentences and a supplementary table comparing baseline characteristics of responders and non-responders. That is the minimum, and it is not sufficient. A study reporting well would provide: cumulative retention from baseline and wave-on-wave rates; a breakdown of non-participation by category — dead, withdrawn, non-contact, refusal, incapacity, other; the number of intermittent responders and the return rate after one and two consecutive misses; the fitted model for response propensity with its predictors and discrimination; the distribution of the resulting weights, including the largest few, because a weight of 40 on one respondent is doing something a reader should know about; and the analytic sample size for the specific estimand in the paper, which is very often much smaller than the wave N. The last of these is the most commonly omitted and the most consequential. Papers routinely report "N = 6,200 at wave nine" and then fit a model requiring complete data on twelve covariates across three waves, on an effective sample of 2,900, without stating it. The retention figure in the abstract describes a sample that was never analysed. The arithmetic of loss is not complicated. The difficulty is entirely in refusing to let a single retention percentage stand in for a structured phenomenon with at least six distinct components, each of which behaves differently and requires a different response. The next chapters take those responses in turn, beginning with the part that is least written about and most determinative: the field infrastructure that keeps a study in contact with people over decades. A note on response rates as a quality metric One habit worth breaking is the treatment of the response rate as a sufficient statistic for data quality. The intuition — higher response rate, less bias — is reasonable and, over the range of variation most studies experience, weak. The reason is arithmetic. Nonresponse bias in a mean is approximately the product of the nonresponse rate and the difference between respondents and non-respondents on the variable in question. A study with 30 per cent nonresponse where non-respondents differ trivially on the outcome has almost no bias. A study with 10 per cent nonresponse where the missing tenth are the sickest people in the cohort has substantial bias on health outcomes. The rate alone cannot distinguish these cases, and a growing body of methodological work comparing surveys with widely different response rates has found the correlation between response rate and nonresponse bias to be surprisingly weak across estimates. This has two implications. The defensive one is that studies should stop presenting a high response rate as evidence of validity, and stop treating a declining one as automatic evidence of decay. The constructive one is that effort should be directed not at the rate but at the differential: at recruiting and retaining the people whose absence would move the estimates. That is the logic of adaptive design, and it sometimes recommends spending less effort on easy cases in order to spend more on hard ones, accepting a lower headline rate in exchange for a less selected sample. Funders and reviewers who evaluate studies on response rate alone make this harder than it should be. None of this licenses complacency about a 40 per cent retention figure. At that level the potential for bias is large whatever the differential, and the range of plausible sensitivity analyses becomes wide enough to undermine most conclusions. The point is narrower: between 70 and 85 per cent, the rate itself is a poor guide, and what matters is who is in the missing fraction. Chapter 3: Retention as Infrastructure The most striking thing about the long-running cohorts with exceptional retention is how little of their success is attributable to anything clever. There is no technique. There is a set of unglamorous operational practices, sustained for decades, by institutions that treated participant contact as core scientific infrastructure rather than as administrative overhead. The Dunedin study, which has followed a birth cohort in New Zealand since the early 1970s, has reported retention in the region of 94 per cent of living participants at its age-45 assessment — a figure that seems almost impossible for a five-decade study. It was achieved by flying participants back to Dunedin from wherever in the world they now lived, by maintaining continuous relationships with family members, by a stable unit with low staff turnover, and by treating each assessment as an event the participant was invited to rather than a survey they were asked to complete. None of that is a trick. All of it is expensive, sustained institutional commitment. This chapter is about that infrastructure: what it consists of, what it costs, and which parts of it are worth defending when budgets are cut. Tracing: the operational heart Everything begins with knowing where people are. In a study running over decades, the majority of participants will move house several times, many will change name on marriage or divorce, some will emigrate, and a growing proportion will move into institutional care. Contact information decays continuously, and a study that does not actively maintain it will discover, at wave seven, that it has lost people it never needed to lose. The core insight of good tracing practice is that it is cheaper to maintain contact than to restore it. A participant whose address the study has always known costs a stamp. A participant last located eight years ago costs hours of skilled searching with a materially lower success rate. This drives the whole design of tracing: continuous, low-intensity, redundant. The practical components, roughly in order of cost-effectiveness: Multiple contact points collected at every wave. Not one address and one phone number, but a home address, a mobile number, an email address, a social media handle where the participant is willing, and — critically — the names and contact details of two or three "stable contacts": people likely to know the participant's whereabouts even if the participant moves. Traditionally a parent, sibling or close friend not living in the same household. Stable contacts are the single highest-yield tracing resource most studies have, and they cost thirty seconds of interview time to collect and a minute to verify. Between-wave contact. A birthday card, a newsletter, a change-of-address freepost card, a short "keeping in touch" mailing. These serve two purposes: they maintain the relationship, and they generate address failures that the study learns about immediately rather than at the next wave. Returned mail is information. A study that only contacts people at data collection waves finds out about every move at the worst possible moment. Administrative and commercial address services. National change-of-address registers, electoral registers where accessible, credit-reference and consumer databases, and in some jurisdictions health service registration files. These have variable coverage and are subject to legal and ethical constraints that differ sharply by country. Where a study can secure consent at baseline for periodic address updates from a national health registration system — as some UK studies have — it converts a hard problem into an easy one. Record linkage for vital status. Regular linkage to death registration is not only substantively valuable but operationally essential: a study that does not know who has died will waste effort tracing them and, worse, will misclassify death as refusal in its attrition analysis. Skilled manual tracing. For the residual hard cases, experienced tracers using open sources, social media, telephone directories, and contact with the stable contacts. This is labour-intensive and has a real yield — studies that maintain a dedicated tracing function typically recover a substantial minority of otherwise-lost cases — but it is the most expensive per case and should be the last line, not the first. Table 3 summarises the tracing toolkit. Table 3. Tracing methods by cost, typical yield and best use. Method Relative cost per case Typical yield Best used Multiple contact details at interview Very low Prevents most losses Every wave, without exception Stable contacts (relatives, friends) Very low High for movers Collect at baseline, refresh every wave Between-wave mailings Low Early warning of moves Annually or more often National address/health registers Low to moderate Very high where available Where consent and law permit Commercial address databases Moderate Moderate, variable Batch searches before fieldwork Death and hospital record linkage Low once established Definitive on vital status Continuously Skilled manual tracing High Moderate on residual cases Last resort, hard cases only Note: costs and yields vary enormously by country, legal framework and study population; the ordering is more stable than the levels. The relationship, and what it is made of Tracing gets a study to the door. What gets it through the door is something less mechanical. Participants in long cohorts routinely describe their involvement in terms that have nothing to do with the study's scientific aims. They describe pride in being part of something long-running, affection for a particular interviewer, a sense of being listened to, curiosity about what the study has found, and — very often — a feeling of obligation to a study that has been part of their life since childhood. These are the actual currencies of retention, and they can be cultivated or squandered. Interviewer continuity is the most reliably documented operational lever. Being visited by the same interviewer wave after wave substantially raises the probability of participation. The mechanism is not mysterious: an established interviewer knows the household, knows how to approach it, has credibility at the doorstep, and has a personal relationship that makes refusal socially costly in a mild and legitimate way. Studies that can assign interviewers to cases across waves should do so; studies that use commercial fieldwork agencies with high turnover lose this advantage and pay for it in response rates. The cost of interviewer continuity is scheduling inflexibility and vulnerability to a single person's departure; the benefit is typically larger. Feedback to participants is the second lever, and the most underused. Participants who receive something back — study newsletters, personalised results where clinically appropriate, findings summaries, occasional media coverage they can see themselves in — report higher willingness to continue. Feedback of individual clinical results is ethically complex and operationally demanding, and studies differ on how much to give, but the general principle that a one-way extraction of information degrades a relationship is not seriously disputed. Visible institutional stability matters over long horizons. Participants who have been contacted by four different universities under three different study names over twenty years have no institution to be loyal to. Studies that maintain a consistent name, a consistent visual identity, a permanent freephone number and a website that has not been allowed to rot are investing in something real. Respect at the point of contact is the hardest to systematise and the easiest to lose. Interview length, question sensitivity, the tone of reminder letters, whether a request to be contacted in the evening is honoured, whether a participant who says "not this year" is treated as a refusal or as a deferral. Most permanent refusals in panels are not principled objections to research; they are the accumulated residue of small frictions. Refusal conversion, and its limits When a participant declines, studies face a decision about whether and how to re-approach. The evidence is that conversion efforts work — a meaningful fraction of soft refusals can be converted, particularly by a different interviewer, with a shorter instrument, or after an interval — and that the converted respondents are systematically different from those who complied immediately, which is exactly why they are worth recovering. The methodological argument for conversion is that it directly reduces the nonresponse bias rather than merely increasing sample size. Converting the people most similar to permanent non-respondents shifts the responding sample towards the full sample, whereas adding more easily-obtained respondents does not. This is the logic behind responsive and adaptive survey designs, which allocate effort dynamically towards cases underrepresented in the achieved sample rather than towards cases that are simply easiest to obtain. Two limits are worth being honest about. First, conversion effort has sharply diminishing returns, and there is a point past which further contact is harassment. Studies should have a written rule about maximum contact attempts and about the treatment of firm refusals, both for ethical reasons and because a badly handled conversion attempt converts a soft refusal into a permanent one. Second, the participants hardest to convert are the ones whose absence biases the study most, so the residual after a good conversion programme is still selected — just less so. Mode and the comparability trade Offering more ways to respond — face-to-face, telephone, web, paper, app — raises response rates. It also introduces mode effects, which are real, sometimes large, and directionally predictable. Self-completion modes (web, paper) reduce social desirability bias relative to interviewer-administered modes. People report more alcohol, more drug use, more risky sexual behaviour, and more symptoms of depression to a screen than to a person. Interviewer-administered modes produce better completion of complex sequences, more accurate income reporting with prompting, and lower item non-response. Telephone interviews truncate response scales relative to face-to-face. Web respondents satisfice more on long grids. For a cross-sectional study, mode effects are a nuisance. For a longitudinal study they are a threat to the central estimand, because a change in mode between waves produces an apparent change in the outcome that is entirely artefactual. A cohort that moved from face-to-face to web between 2018 and 2021 — as many did under pandemic pressure — will see a jump in reported depressive symptoms that has nothing to do with the pandemic and everything to do with the absence of an interviewer. There are three defences, and studies should use all of them. Overlap designs: for one wave, randomise a subsample to each mode, so the mode effect can be estimated and adjusted. Within-person anchoring: exploit people who happen to switch modes at different times to separate mode from period. Mode indicators in every model: record mode at the observation level and carry it into analysis as a covariate, never as an afterthought. The deeper point is that mode expansion is a trade, not a free improvement. It buys response rate at the cost of measurement comparability. For a study whose estimand is within-person change, that trade can be a bad one, and there are circumstances in which holding the mode constant and accepting a lower response rate is the better scientific decision. Very few studies frame it this way, because response rate is a visible operational metric and mode effect is an invisible measurement one. Budgeting for retention Retention infrastructure competes for money with measurement. A study can spend its marginal pound on a new biomarker assay or on a tracing officer, and the assay is far easier to justify in a grant application because it produces a visible new variable. Three arguments push the other way. First, the assay is useless on people who are not there; retention multiplies the value of every measure the study takes. Second, retention spending is cumulative and compounding — a participant retained this wave is available for every future wave — while a one-off measure is a one-off. Third, retention spending is hard to restart. A study that cuts its between-wave contact programme for three years to save money does not merely lose three years of contact maintenance; it loses participants whose addresses have gone stale and whose sense of connection has faded, and recovering them costs far more than the saving. The practical recommendation is to protect a fixed proportion of the study's core budget for participant relationship and tracing, ring-fenced from the measurement budget, and to treat proposals to cut it with the same scepticism as a proposal to stop measuring the primary outcome. In effect, the contact database is a scientific asset with a depreciation schedule, and studies that fail to maintain it are running down capital while reporting a balanced budget. What infrastructure cannot do It would be dishonest to end a chapter on retention without saying plainly that the ceiling is real. Some people will not participate, at any price, with any interviewer, in any mode. Some will become too ill. Some will emigrate beyond the reach of tracing. Some will die. In an ageing cohort, the proportion who cannot participate rises inexorably regardless of how good the field operation is, and that proportion is not random — it is concentrated precisely among the people whose health trajectories the study most wants to observe. This is why the second half of this booklet exists. Excellent fieldwork reduces the size of the missing-data problem and, more importantly, changes its character: it converts non-contact, which is largely ignorable given mobility predictors, into a residual of refusal and incapacity, which is not. A study with 90 per cent retention has a small and probably nasty missingness problem. A study with 55 per cent retention has a large one whose nastiness is harder to characterise. Both need analytic adjustment; the first needs far less of the analyst's faith. Proxy respondents and the incapacity frontier One operational response to incapacity deserves separate discussion, because it sits exactly on the boundary between fieldwork and measurement. When a participant becomes too ill or too cognitively impaired to respond, some studies collect data from a proxy — a spouse, adult child or care home staff member. The Health and Retirement Study and several other ageing panels have used proxy interviews extensively, and the practice materially reduces the differential loss of the sickest participants. Without it, the very people whose decline the study exists to document vanish from the data precisely when they become interesting. Proxy data is not equivalent data, and pretending otherwise causes problems. Proxies report functional limitations and observable behaviours reasonably well; they report pain, mood, cognition and subjective wellbeing poorly and with systematic direction — proxies tend to overstate impairment and understate wellbeing relative to the person's own report. A study that switches a declining participant from self-report to proxy report will therefore record a step change in measured impairment at the moment of switching, partly real and partly artefactual, and an analysis that ignores the switch will attribute the whole step to disease progression. The defences are the same as for mode effects and are rarely applied with the same rigour: record proxy status at the observation level, include it in every model, and where possible collect both self and proxy reports during an overlap period so the gap can be estimated. Studies that have done this find the gap large enough to matter for trajectory estimation. The wider point is that the incapacity frontier is where a study's retention and its measurement collide most directly. Every mechanism for keeping frail participants in the study — proxies, shortened instruments, telephone substitution for clinic visits, care home visits — buys coverage at the price of comparability. Refusing all of them buys comparability at the price of a survivor-selected sample. There is no option that gives both, and the studies that handle it best are those that choose coverage, document the compromise obsessively, and give analysts the variables needed to model it. Hashtags: #DesigningLongitudinalStudies #LongitudinalResearch #CohortStudies #PanelStudies #ParticipantRetention #Attrition #MissingData #PanelConditioning #WaveNonresponse #IntermittentMissingness #MultipleImputation #MissingAtRandom #MissingNotAtRandom #InverseProbabilityWeighting #CensoringWeights #SurvivorshipBias #ResponsePropensity #AuxiliaryVariables #ParticipantTracing #RetentionInfrastructure #ModeEffects #MixedModels #RecordLinkage #LongitudinalEstimands #FutureOfLongitudinalResearch

Download the Book (PDF): Introduction In 1999 the Māori scholar Linda Tuhiwai Smith, of Ngāti Awa and Ngāti Porou descent, published Decolonizing Methodologies: Research and Indigenous Peoples. Its opening pages contain a sentence that has been quoted in thousands of ethics seminars since: "The word itself, 'research', is probably one of the dirtiest words in the indigenous world's vocabulary." Smith's book, now in its third edition, did not argue that Indigenous peoples should reject inquiry. It argued the opposite. Indigenous communities had always asked questions, tested answers, and handed knowledge down with care. What they had learned to distrust was a particular arrangement of research, one in which outsiders arrived with their own questions, took away blood, bones, stories, plants and photographs, published under their own names, and left the people they had studied with little except a sense of having been used. This booklet borrows the phrase in its title as a mark of debt, not as a claim to stand beside Smith's work. Her book remains the foundation, and anyone who has not read it should. What follows is narrower and more practical. It asks what has actually changed in the quarter century since, especially in the sciences that trade in biological material and data, and what still has to change for research with Indigenous peoples to stop reproducing the pattern Smith described. The argument The controlling claim of this booklet is simple to state and hard to carry out. Decolonizing research is not an ethical add-on to otherwise unchanged science. It is a transfer of decision-making authority, over which questions get asked, who holds the samples and data, how knowledge is described and reused, who is credited, and where the benefits go, from research institutions to the Indigenous peoples whose lives, lands and bodies are the subject of the work. Good intentions do not accomplish that transfer. It happens, or fails to happen, in consent forms, data access committees, material transfer agreements, repository metadata, authorship policies, grant budgets and national law. That claim has a practical consequence. A researcher who wants to work well with Indigenous communities needs more than sensitivity. She needs to know what the CARE Principles for Indigenous Data Governance require, how the Nagoya Protocol and the 2024 WIPO treaty on genetic resources shape her obligations, what a Traditional Knowledge Label does in a repository record, how a tribal research review board differs from a university one, and how to write a budget that pays community researchers at the rates she would pay a postdoctoral fellow. Much of the booklet is devoted to that working knowledge. Why now Three developments make the subject urgent in 2026. The first is genomics. The push to diversify genetic databases, which were overwhelmingly built from people of European ancestry, has turned Indigenous DNA into a scarce and valuable research resource. Funders now reward projects that recruit underrepresented populations. That pressure can produce real benefit, but it also recreates the conditions of the Havasupai case, in which blood given for one purpose was quietly used for others. The rules of open data, designed to make science faster and fairer, can make things worse for groups whose genomes, once deposited, can be downloaded by anyone in the world. The second is the commercialisation of digital sequence information. Genetic sequences from plants, animals and microbes collected on Indigenous lands can now be stored, shared and used to design products without anyone touching a physical sample. The access and benefit-sharing system built on the Convention on Biological Diversity assumed that benefits would follow physical material across borders. In 2024 the parties to that convention created the Cali Fund to try to capture some value from digital sequences, with at least half of its money earmarked for Indigenous peoples and local communities. Whether that mechanism works is one of the open questions of the decade. The third is institutional. Journals, funders and universities have begun to write the language of equity into their rules. Nature Portfolio asked authors from 2022 onward to address local participation and benefit in an inclusion and ethics statement, citing the San Code of Research Ethics. Canada, Australia and Aotearoa New Zealand have Indigenous research ethics codes with real force. Tribal nations in the United States run their own review boards. Yet a large gap remains between policy and practice. Many researchers still treat community engagement as a stage to be completed before the real science starts. Scope and terms The booklet concentrates on research with Indigenous peoples in settler-colonial states such as the United States, Canada, Australia and Aotearoa New Zealand, and on Indigenous and local communities in southern Africa, the Amazon and elsewhere where cases are well documented. The principles often apply to other marginalised communities and to research partnerships between institutions in rich and poor countries, and the text draws on that wider literature where it helps. But the booklet does not claim that every inequity in global science is identical to colonial dispossession. The specific claims of Indigenous peoples rest on their prior occupation of land, their continuing political existence as peoples, and rights recognised in the 2007 United Nations Declaration on the Rights of Indigenous Peoples. A note on language. "Indigenous" is capitalised as a proper term for peoples, as most Indigenous organisations prefer. Where a specific nation or people is involved, the booklet uses their name. "Community" is a convenient word that can hide real differences, since a tribal government, a council of Elders, a family group and an urban Indigenous organisation may not agree, and the question of who speaks for a community is one of the hardest in this field. The booklet returns to it more than once. The booklet also avoids a common shortcut. It does not treat decolonization as a metaphor for any reform a university might like to make. The Unangax̂ scholar Eve Tuck and her co-author K. Wayne Yang warned in 2012, in an essay titled "Decolonization is not a metaphor", that the word loses its meaning when it is stretched to cover diversity training or curriculum refreshes while land, resources and authority stay where they were. Research is not land, but the same test applies. A change counts here if it moves real control over material, data, knowledge, credit or money toward the people concerned. A change that only alters vocabulary does not. How the booklet is organised Chapter 1 sets out the extractive inheritance: the history of collecting and the cases that made research a dirty word, from the Yanomami blood samples to the Havasupai lawsuit and the patenting of Hoodia. Chapter 2 examines sovereignty as a research principle and the data governance frameworks that give it shape, including OCAP, CARE and the principles of Te Mana Raraunga. Chapter 3 turns to genomes and biobanks, where the conflict between open science and collective control is sharpest. Chapter 4 covers the law of biodiversity and benefit-sharing, from the Nagoya Protocol to the Cali Fund and the WIPO treaty. Chapter 5 asks what it means to bring Indigenous knowledge systems into scientific work without absorbing and erasing them. Chapter 6 describes the practical infrastructure of labels, notices and repository practice. Chapter 7 deals with credit and authorship. Chapter 8 puts the pieces together in the design of a reciprocal project from first conversation to closure. The Conclusion argues for what should follow. The booklet is written for scientists, students, ethics reviewers, research managers, funders and Indigenous readers who want to see how the machinery works. It does not speak for any Indigenous people. Where it describes Indigenous frameworks, it relies on what their authors have published, and it points readers to those authors. Chapter 1. The Extractive Inheritance Every research relationship with an Indigenous community begins with a history the researcher may not know but the community certainly does. A geneticist arriving in a village in 2026 is, in local memory, the latest in a line that includes the collector who took skulls from a burial ground, the anthropologist who photographed ceremonies that were not meant to be seen, the missionary who recorded a language while forbidding children to speak it, and the survey team that took blood and never came back. The researcher's personal decency does not erase that line. Trust is not a starting condition; it has to be built against a record. This chapter describes that record, not as a catalogue of villains but as a set of patterns. The patterns matter because they recur. Each of the famous cases shares a structure: a gap between what a community understood it was agreeing to and what the researchers did, a movement of material or knowledge out of community control, and a distribution of credit and benefit that ran almost entirely one way. Collecting as a scientific method Nineteenth and early twentieth century science treated Indigenous peoples as objects of collection. Museums in Europe, North America and Australia acquired tens of thousands of human remains, many taken from graves, battlefields and hospitals. Physical anthropologists measured skulls to build racial typologies. Samuel George Morton's collection in Philadelphia and the vast holdings of the Smithsonian Institution were assembled on the premise that Indigenous peoples were vanishing and that their bodies, like their artefacts, belonged to the record of humanity rather than to their descendants. The same logic governed "salvage" ethnography. Researchers recorded songs, stories, and ceremonial knowledge on the assumption that the cultures producing them would soon disappear. The recordings were deposited in archives, catalogued under the collector's name, and made available to anyone with a reader's ticket. Communities that survived, as most did, later found their sacred material in public collections, sometimes described with errors, sometimes available for commercial licensing. The legal response came late. The United States passed the Native American Graves Protection and Repatriation Act in 1990, requiring federally funded museums to inventory Native American human remains and cultural items and to consult with tribes on their return. Progress was slow for decades, and in January 2024 revised federal regulations took effect requiring museums to obtain free, prior and informed consent before exhibiting or researching covered items. Several major museums closed galleries in response. Australia, Canada and Aotearoa New Zealand run their own repatriation programmes, and many remains have come home, though many have not. The point of recalling this history in a book about present-day science is that the underlying assumption did not die with the skull collections. The assumption was that material taken from Indigenous people becomes the property of science, to be kept, studied and reused indefinitely at the discretion of researchers. That assumption survives in the freezers of biobanks and the servers of sequence databases. Blood that did not come home In the late 1960s a team led by the American geneticist James Neel, working with the anthropologist Napoleon Chagnon, collected blood from Yanomami people in the Amazon rainforest of Venezuela and Brazil. The samples were shipped to laboratories in the United States and stored for decades. The Yanomami had not given informed consent in any modern sense. Accounts describe blood exchanged for trade goods such as machetes and cooking pots. Crucially, Yanomami funerary practice requires the destruction of a dead person's bodily remains and possessions, and many of the people who had given blood had since died. Their blood sitting in freezers in Pennsylvania was, from the Yanomami point of view, a continuing violation. The Yanomami leader Davi Kopenawa campaigned for the return of the samples for more than a decade. In April 2015, 2,693 samples held by Pennsylvania State University were returned to Brazil and received by Yanomami communities in a ceremony. Kopenawa's summary is often quoted: the Americans had taken the blood without saying anything in the Yanomami language about the tests they would do. The case illustrates three features that recur. The consent process, where one existed, was not in the participants' language or terms. The samples outlived the people who gave them and the purposes for which they were taken. And the institutions holding them treated their return as a concession rather than an obligation. The Havasupai case The case that did most to change genetic research ethics in the United States involved the Havasupai Tribe, whose reservation lies within the Grand Canyon in Arizona. Between 1990 and 1994 researchers from Arizona State University, among them the geneticist Therese Markow and the anthropologist John Martin, collected blood samples from around 400 tribal members. The Havasupai had asked for help with a devastating epidemic of type 2 diabetes. They understood the study to be about diabetes. The samples were later used for research on schizophrenia, on inbreeding, and on the population's migration history. Those uses were not trivial deviations. Studies of inbreeding touched on taboo subjects. Research on ancient migration, suggesting ancestral origins in Asia, contradicted the Havasupai account of their origin in the canyon itself, a matter central to their identity and, potentially, to land claims. The tribe learned of the secondary uses in 2003, partly through a tribal member, Carletta Tilousi, who attended a university presentation. Dissertations and papers had been produced from the samples, and the samples had been shared with other researchers. The tribe sued the Arizona Board of Regents in 2004. In April 2010 the parties settled. The university agreed to pay $700,000 to 41 tribal members, to return the remaining blood samples, and to provide other forms of assistance. Michelle Mello and Leslie Wolf, writing in the New England Journal of Medicine that year, drew the lessons for institutional review boards and biobanks: broad consent language can be legally defensible and still be experienced as betrayal, and group harms, harms to a community's reputation, beliefs and political interests, fall outside the individual-focused frame of standard research ethics. That last point deserves emphasis. United States research regulation, set out in the Common Rule, protects individual participants. It asks whether the individual consented and whether the individual faces risk. It has no category for harm to a people. The Havasupai injury was not primarily that any one person's privacy was breached, but that the tribe's collective account of itself was challenged, and its members stigmatised, through research it had not approved. A consent form signed by each of 400 people could not have authorised that, because the interest at stake did not belong to any of them individually. Hoodia and the question of who owns knowledge The third pattern concerns knowledge rather than bodies. San peoples of southern Africa have long chewed Hoodia gordonii, a succulent plant, to suppress hunger and thirst on long hunting trips. In the 1960s and 1970s South Africa's Council for Scientific and Industrial Research (CSIR) investigated the plant, and in the 1990s it patented an appetite-suppressing compound, later named P57, and licensed it to the British company Phytopharm, which in turn entered a development agreement with Pfizer. The San had not been consulted. A statement attributed to the licensing company at the time suggested that the San who had discovered the plant's properties no longer existed as a people. After campaigning by San organisations and their lawyers, the South African San Council and the CSIR signed a benefit-sharing agreement in 2003 under which the San would receive a share of royalties. The drug never reached the market as a pharmaceutical, so the money that flowed was modest. But the case set a precedent. It showed that traditional knowledge could be the uncredited foundation of a patent, that the holders of that knowledge could be written out of the story by the claim that they had disappeared, and that negotiation after the fact could at least partly correct the record. The later rooibos agreement, discussed in Chapter 4, built on that precedent. The Havasupai case was not unique. In the 1980s the geneticist Ryk Ward, then at the University of British Columbia, collected blood from several hundred members of the Nuu-chah-nulth First Nations on Vancouver Island for a study of rheumatoid arthritis, a disease that weighed heavily on their communities. The study did not find the genetic link it sought. Ward later moved to the University of Utah and then to Oxford, and the samples went with him. Papers followed on mitochondrial DNA and the ancient population history of the Americas, a subject the Nuu-chah-nulth had never agreed to. When the Nations learned of these uses they asked for the samples back, and a return was eventually negotiated in the early 2000s. The case became a teaching example in Canada and fed directly into the chapter on research involving First Nations, Inuit and Métis peoples in the national Tri-Council Policy Statement, first issued in 2010. Why a better consent form is not enough The instinctive institutional reaction to these cases has been to improve consent. Forms are now longer, secondary uses are spelled out, and participants are offered tick boxes for future research. That is progress, but it misses the lesson of the cases in three ways. First, consent is given at one moment and governs an open-ended future. A form signed in 1991 cannot anticipate whole-genome sequencing, polygenic scores, ancestry inference or machine learning. The Havasupai participants could not have consented to analyses that did not yet exist, and a modern participant cannot consent to analyses that will exist in 2040. The only way to give people a say over future uses is to create a standing body, accountable to them, that decides as those uses arise. Second, individual consent aggregates individual interests. It cannot protect interests that belong to a group. If 400 people each consent to a study of their ancestry, the resulting paper still speaks about all Havasupai, including those who refused and those not yet born. Several Indigenous ethics frameworks respond by requiring collective consent, given by a tribal government or community body, in addition to individual consent. The two are not alternatives. A tribal council's approval does not force any individual to participate, and an individual's willingness does not override the council's refusal. Third, consent does nothing to change who holds the material. A perfectly consented sample stored in a university freezer, governed by a university committee, remains under university control. The structural question of this booklet is not whether people agreed but who decides. That is why the chapters that follow concentrate on governance, law, infrastructure and money rather than on paperwork. Helicopters and parachutes Collecting, secondary use and unacknowledged knowledge are the three historical patterns. A fourth, more recent, is the structure of research partnerships themselves. "Helicopter research", sometimes called "parachute science", describes projects in which researchers from well-resourced institutions fly into a community or country, collect samples or data, and leave to analyse and publish at home, with local collaborators serving as guides, fixers or sample collectors rather than intellectual partners. The pattern is measurable. Studies of authorship in global health journals have repeatedly found that papers reporting research conducted in low- and middle-income countries often have first and last authors based in high-income countries, and that local researchers are underrepresented in the positions that signal intellectual leadership. The same applies within settler states. Papers on Indigenous health, genetics and ecology have frequently had no Indigenous authors at all. Helicopter research is not always the product of bad faith. It flows from incentives. Grants are held by the institution that applies. Timelines are set by funding cycles, not by community decision-making. Promotion committees count first-author papers. Data become valuable when they are pooled and analysed by people with computing resources. A researcher who wants to do otherwise must swim against each of these currents, which is why this booklet insists that the problem is structural. The European Union-funded TRUST project, which ran from 2015 to 2018, produced the Global Code of Conduct for Research in Resource-Poor Settings in 2018, organised around the values of fairness, respect, care and honesty. Its first article states that local relevance of research should be determined in collaboration with local partners. The code, together with the San Code of Research Ethics developed alongside it, later shaped the European Commission's guidance for Horizon Europe applicants and the Nature Portfolio policy on helicopter research introduced in 2022. Two habits of language keep the older logic alive inside modern papers. The first is the description of Indigenous groups as "isolated populations" or "genetic isolates", valuable because of their supposed separation from the rest of humanity. The framing is sometimes technically accurate for a narrow purpose, but it repeats the salvage idea that a people's worth to science lies in its difference and its fragility. The second is the language of resource: a community "provides a unique resource" for mapping disease variants, or its knowledge is "a resource" for drug discovery. The people become a deposit, and the research becomes mining. There is also a cost that rarely appears in ethics applications: research fatigue. Some communities have been studied intensively for generations. Inuit communities in the Canadian Arctic, Aboriginal communities in the Northern Territory and Navajo communities in the American Southwest have hosted wave after wave of surveys, trials, sampling campaigns and graduate theses, often on overlapping questions, with results rarely returned in usable form. Every new project draws on the time of the same Elders, health workers and interpreters. A research team that treats its visit as a single event, rather than as one more demand on a finite local capacity, misreads the situation from the start. What the patterns teach It is tempting to read these cases as stories of individual misconduct, corrected by better rules. That reading is partly right. The Havasupai researchers and the CSIR made choices that were questionable even by the standards of their time. But the more important lesson is that each case was possible because authority over the material, the data or the knowledge sat entirely with the research institution. The Yanomami had no say over where their blood was stored. The Havasupai had no seat on the committee that approved new uses. The San had no standing in the patent process. Linda Tuhiwai Smith made this point in structural terms. Research, she argued, is implicated in the worst excesses of colonialism because it produced the knowledge that colonisers used to classify, manage and dispossess. The remedy she proposed was not to stop research but to reorient it around Indigenous agendas: self-determination, survival, recovery and development. Her book lists twenty-five Indigenous research projects, from claiming and testimony to restoring, returning and protecting, each an example of research done by Indigenous people for their own purposes. For the scientist, the practical lesson is this. The history is not background. Communities will judge any new proposal against it, and they are right to do so, because the structures that produced the old harms are still largely in place. A researcher who begins by acknowledging that history, and by explaining concretely what will be different this time and who will have the power to enforce it, starts on firmer ground than one who begins with reassurances. A short test Readers who want to apply this chapter to their own work can ask four questions of any project involving Indigenous participants, samples, lands or knowledge. • Who decided what question the project would ask, and could the community have said no to the question itself? • Where will the material and data be held in ten years, and who will be able to authorise new uses? • Whose knowledge does the project rely on, and how will that reliance be recorded and credited? • If the project produces something of value, a drug, a publication, a patent, a policy, a career, how will that value be shared, and who decided the split? If the answer to each question is "the research team", the project is operating within the extractive inheritance, however kind its members. The following chapters describe the tools that exist to give different answers. Chapter 2. Sovereignty as a Research Principle The word that separates Indigenous research ethics from ordinary research ethics is sovereignty. Ordinary ethics asks whether a study treats participants well. Sovereignty asks who has the authority to decide whether the study happens at all, on what terms, and what becomes of its products. The first question can be answered by a university committee. The second, where Indigenous peoples are concerned, cannot. This chapter traces how the political claim to self-determination became a set of working principles for research and data. It covers the international foundation in the United Nations Declaration on the Rights of Indigenous Peoples, the tribal and national review bodies that exercise research authority, and the data sovereignty frameworks, OCAP, CARE, the principles of Te Mana Raraunga and Maiam nayri Wingara, that translate authority into rules a data manager can follow. From self-determination to research authority The United Nations General Assembly adopted the Declaration on the Rights of Indigenous Peoples on 13 September 2007, with 144 states in favour and four against. The four were Australia, Canada, Aotearoa New Zealand and the United States, the settler states with the largest stakes. All four later reversed their positions, and Canada went further, passing the United Nations Declaration on the Rights of Indigenous Peoples Act in 2021 to align its federal laws with the Declaration. The Declaration is not a treaty and does not bind states in the way a convention does. It matters for research because of what it recognises. Article 3 affirms the right of Indigenous peoples to self-determination, by virtue of which they "freely determine their political status and freely pursue their economic, social and cultural development". Article 18 affirms the right to participate in decision-making on matters affecting them. Article 31 is the most directly relevant. It recognises the right of Indigenous peoples to maintain, control, protect and develop their cultural heritage, traditional knowledge and traditional cultural expressions, including "human and genetic resources, seeds, medicines, knowledge of the properties of fauna and flora". The same article recognises their right to intellectual property over that heritage and knowledge. Read together, these articles say that research on Indigenous genetic resources and knowledge is a matter within Indigenous self-determination. That does not mean every study needs the approval of every Indigenous person. It means that Indigenous peoples, through their own institutions, have a legitimate claim to decide whether and how such research proceeds, and that states and institutions should respect that claim. The Declaration also establishes the standard of free, prior and informed consent, which appears in several articles, most prominently in relation to land and resources. Research ethicists have borrowed the term, but its meaning in the Declaration is collective. Consent is given by a people through its representative institutions, freely, before the activity begins, and with full information. It can be withheld. A consultation that proceeds regardless of the answer is not consent. Who holds the authority In practice, research authority is exercised through several kinds of body. In the United States, the 574 federally recognised tribes are sovereign nations with governmental authority over their lands and members. Many exercise that authority over research through tribal research review boards or through tribal council resolutions. The Navajo Nation Human Research Review Board, established in the 1990s, reviews all research conducted on the Navajo Nation or involving its members. In 2002 the Navajo Nation Council placed a moratorium on genetic research, pending amendment of the Nation's health research code. That moratorium remains in force. A genetics policy working group consulted widely in 2018 and 2019, and a 2024 council resolution re-established the group to complete the code amendments needed before the moratorium can be lifted. The Navajo case shows that a moratorium is not a refusal to engage with science. It is an exercise of authority, maintained until the Nation has rules it trusts. Other tribes approve research through specific agreements. The Cherokee Nation runs its own institutional review board. The Cheyenne River Sioux Tribe hosts the Native BioData Consortium, an Indigenous-led biorepository discussed in Chapter 3. Researchers working with tribes in the United States must therefore expect two levels of review: their own university's board, governed by federal regulation, and the tribe's, governed by tribal law. The tribal review is not an extra hurdle to be cleared after the university approves. It is a separate sovereign decision. In Canada, the Tri-Council Policy Statement, the ethics framework for all federally funded research, has since 2010 contained a chapter on research involving First Nations, Inuit and Métis peoples. It requires community engagement where research is likely to affect a community's welfare, recognises the role of community governance in decisions about participation, and addresses collective interests in data and biological materials. In Australia, the Australian Institute of Aboriginal and Torres Strait Islander Studies issued its Code of Ethics for Aboriginal and Torres Strait Islander Research in 2020, organised around Indigenous self-determination, leadership, impact and value, and sustainability and accountability. The National Health and Medical Research Council's 2018 guidance sets out six core values: spirit and integrity, cultural continuity, equity, reciprocity, respect and responsibility. In Aotearoa New Zealand the Health Research Council's Te Ara Tika guidelines apply Māori ethical concepts to research review. These frameworks differ in force and detail, but they share a structure. They recognise collective interests alongside individual ones, they expect Indigenous governance to be involved in decisions, and they treat benefit and reciprocity as obligations rather than courtesies. Data sovereignty Research authority over the conduct of a study is only part of the problem. Studies produce data, and data travel. A survey conducted with a community's full approval may yield a dataset that is later merged with others, reanalysed by strangers, or used by governments to design policies the community opposes. Indigenous data sovereignty is the claim that Indigenous peoples have rights and interests in data about them, their lands and their resources, wherever those data are held. The term gained currency through the work of scholars and practitioners in Aotearoa New Zealand, Australia, Canada and the United States in the mid-2010s. Tahu Kukutai and John Taylor's edited volume Indigenous Data Sovereignty: Toward an Agenda, published by ANU Press in 2016, brought the arguments together. Networks followed: Te Mana Raraunga, the Māori Data Sovereignty Network, founded in 2015; the United States Indigenous Data Sovereignty Network in 2016; Maiam nayri Wingara, the Aboriginal and Torres Strait Islander data sovereignty collective in Australia, whose principles were set out at a national summit in 2018; and the Global Indigenous Data Alliance, launched in 2019. Table 1 compares the main frameworks that emerged. They are not rival standards. Each reflects the legal and cultural circumstances of the people who wrote it, and most practitioners use them together. Table 1. Principal Indigenous data governance frameworks. Framework Origin Core elements Typical use OCAP First Nations, Canada; 1998 (First Nations Information Governance Centre) Ownership, Control, Access, Possession Health surveys, First Nations data agreements Te Mana Raraunga principles Māori Data Sovereignty Network, Aotearoa; 2018 Rangatiratanga, whakapapa, whanaungatanga, kotahitanga, manaakitanga, kaitiakitanga Government and research data involving Māori Maiam nayri Wingara principles Aboriginal and Torres Strait Islander summit, Australia; 2018 Control, context, relevance, accountability, protection Australian government and research data CARE Principles Global Indigenous Data Alliance; 2019, published 2020 Collective benefit, Authority to control, Responsibility, Ethics Repositories, open data, genomics, biodiversity OCAP: ownership as a starting point OCAP, which stands for ownership, control, access and possession, was coined in 1998 during the development of the First Nations Regional Health Survey, a national survey designed and governed by First Nations. It is now a registered trademark of the First Nations Information Governance Centre, which offers training in its application. Ownership means that a community or group owns its cultural knowledge, data and information collectively, in the way an individual owns personal information. Control means that First Nations have the right to control all aspects of research and information management that affect them, from conception to completion. Access means that First Nations must have access to information about themselves, wherever it is held, and the right to make decisions about access by others. Possession refers to physical control of data, and is the mechanism by which ownership is protected. The strength of OCAP is its bluntness. It says plainly that data about First Nations belong to First Nations and that possession matters. A dataset held on a university server under a sharing agreement is controlled, in practice, by the university. A dataset held on a First Nations server, with the university granted access, is controlled by the Nation. That difference sounds administrative. It is the whole question. Māori principles and relational obligations The principles published by Te Mana Raraunga in October 2018 express data sovereignty through Māori concepts. Rangatiratanga, or authority, affirms Māori rights to control data and to exercise that control through their own institutions. Whakapapa, or genealogy, recognises that data have a context and a lineage, and that disaggregating Māori data from their origins can distort their meaning. Whanaungatanga, or obligations of kinship, recognises that individuals' rights in data are balanced by responsibilities to the collective. Kotahitanga, or collective benefit, requires that data ecosystems support Māori wellbeing. Manaakitanga, or reciprocity, requires that data use uphold the dignity of Māori communities. Kaitiakitanga, or guardianship, frames data as a taonga, a treasure, to be held and protected by those with a duty to do so. The Māori framing adds something the more procedural frameworks lack. It treats data not as inert records but as things with relationships. A genome is whakapapa made visible; it connects a person to ancestors and descendants. A dataset on a river's health is connected to the people whose identity is bound up with that river. Rules about access and use follow from those relationships rather than from abstract rights of ownership. CARE: a complement to FAIR The framework most likely to reach a laboratory scientist is CARE. The Global Indigenous Data Alliance released the CARE Principles for Indigenous Data Governance in 2019, and Stephanie Russo Carroll and colleagues described them in the Data Science Journal in 2020. The acronym is a deliberate reply to FAIR, the principles published by Mark Wilkinson and colleagues in Scientific Data in 2016, which hold that research data should be findable, accessible, interoperable and reusable. FAIR is about data as objects: how they should be described and stored so that machines and people can find and use them. It says nothing about who should benefit or who should decide. CARE supplies what is missing. Collective benefit means that data ecosystems should be designed so that Indigenous peoples benefit from the data, including through inclusive development, improved governance and equitable outcomes. Authority to control affirms Indigenous rights and interests in data and the authority to control them, including through governance of data about lands, territories and resources. Responsibility means that those working with Indigenous data are accountable for showing how their use supports self-determination and collective benefit, including by building capacity and relationships. Ethics means that Indigenous rights and wellbeing should be the primary concern at every stage of the data lifecycle. The slogan that emerged, "Be FAIR and CARE", captures the intended relationship. CARE does not reject open science. It accepts that data should be well described and reusable, but insists that reuse be governed by people with rights in the data. In practice that means, for example, that a FAIR metadata record for an Indigenous dataset should carry information about provenance and permissions, that access may be controlled rather than open, and that repositories should support mechanisms for Indigenous authority. Chapter 6 describes the tools that make this concrete. What sovereignty looks like on paper Principles become real in documents. The most important of these, in most projects, is a research or data governance agreement between the research institution and the Indigenous governing body. Such agreements vary widely, but those that give effect to sovereignty tend to contain a recognisable set of clauses, and a researcher can use the list below as a check on any draft. • A statement that data and biological materials generated by the project are held on behalf of, or owned by, the Indigenous party, with the institution acting as steward rather than owner. • A specified purpose, with any new use, including secondary analyses by the original team, requiring fresh approval from a named Indigenous body. • Rules on location and possession: where data and samples will be stored, whether copies may leave the jurisdiction, and on whose servers. • Rules on access by third parties: whether data may be deposited in public or controlled-access repositories, and who sits on the committee that decides access requests. • Review of outputs before publication, with a defined period, often thirty to ninety days, for the Indigenous party to comment, and a clear statement of what the review covers. Most agreements distinguish between the right to correct factual or cultural errors and to protect sensitive information, which the community holds, and a right to suppress unwelcome findings, which most do not grant. • Arrangements for benefit, capacity and credit, including authorship and acknowledgement. • Provisions for ending the relationship: the return or destruction of samples and data, and the survival of obligations after the grant expires. A university lawyer seeing such an agreement for the first time may object that it gives away institutional property or academic freedom. Both objections deserve a direct answer. The institution does not own data about a people merely because its employees collected them; property in data is a matter of agreement, and the agreement can allocate it differently. Academic freedom protects researchers from interference by their employers and the state in what they may study and say. It does not confer a right to use other people's samples and knowledge without their consent. A researcher remains free to publish critical findings from work she was authorised to do. She is not free to conduct work she was not authorised to do. Tensions within sovereignty Sovereignty frameworks raise hard questions that honest practitioners acknowledge. The first is representation. Who exercises authority for a people? For a federally recognised tribe with an elected council, the answer is formally clear, but councils change, and a council's decision may not reflect the views of all members, including those living off-reservation. Urban Indigenous people, who in several countries are now a majority of the Indigenous population, may have no single governing body. Peoples whose territories cross national borders, such as the San across southern Africa or the Tohono O'odham across the United States and Mexico border, face multiple jurisdictions. A researcher cannot resolve these questions, but she must ask them and must not choose the most convenient interlocutor. The second is the relation between individual and collective rights. An Indigenous person may wish to join a genetic study that her nation's council has declined to approve, or to share her own data with a commercial ancestry company. Most frameworks hold that collective authority governs research conducted within a nation's jurisdiction or that makes claims about the nation, while individuals retain rights over their own participation. The boundary is contested, and the growth of direct-to-consumer genomics makes it more so, since data given individually can yield inferences about whole groups. The third is capacity. Exercising authority over data requires infrastructure: servers, data managers, lawyers, ethics reviewers. Many communities do not have them, and asking them to govern data without resourcing that governance can turn sovereignty into another unfunded burden. The frameworks recognise this, which is why capacity building appears in CARE's principle of responsibility. A funder that endorses Indigenous data sovereignty but does not pay for Indigenous data governance has endorsed a slogan. None of these tensions is a reason to set sovereignty aside. They are reasons to treat it as a practice to be worked out with each partner, rather than a box ticked once a letter of support is on file. Chapter 3. Blood, Genomes and Biobanks Genomics is where the argument of this booklet meets its hardest test. The field has powerful reasons to want Indigenous participation, a strong culture of open data, and a technology that turns a single sample into information about whole lineages. Indigenous peoples have powerful reasons to be cautious: the history set out in Chapter 1, the collective nature of genetic information, and the fact that once a genome is in a public database it cannot be recalled. The question is whether genomic science can be reorganised so that Indigenous participation happens on Indigenous terms. There is now enough experience to say that it can, and to describe what that reorganisation requires. The diversity problem and its trap The case for including Indigenous peoples in genomics is real. In 2016 Alice Popejoy and Stephanie Fullerton reported in Nature that 81 per cent of participants in genome-wide association studies were of European ancestry. Since the genetic architecture of disease differs across populations, and since risk scores built on European data perform worse in other groups, the imbalance means that the benefits of genomic medicine flow disproportionately to people of European descent. Indigenous peoples, who carry high burdens of conditions such as type 2 diabetes, kidney disease and certain cancers, stand to be left out of the precision medicine that genomics promises. Funders responded. The United States National Institutes of Health launched the All of Us Research Program in 2018 with the aim of enrolling a million or more participants, prioritising groups historically underrepresented in research. Similar diversity commitments appear in the plans of national genomics programmes around the world. But the diversity argument has a trap in it, and Indigenous scholars named it early. Keolu Fox, a Kanaka Maoli geneticist, argued in the New England Journal of Medicine in 2020 that inclusion in a programme such as All of Us, on terms set by the programme, risked becoming "the illusion of inclusion". Data would be pooled in a federal resource, available to approved researchers worldwide, including commercial ones, with no mechanism for tribal governance of future use. Krystal Tsosie, Joseph Yracheta and colleagues put the point sharply in the title of a 2021 article in the American Journal of Bioethics: "We have 'gifted' enough". The demand to diversify databases, they argued, repeats the expectation that Indigenous peoples should donate their biological heritage for the good of science, while others control and profit from it. The trap, then, is this. The more valuable Indigenous genomes become to science, the stronger the pressure to recruit Indigenous participants, and the more important it becomes that recruitment be governed by Indigenous authority. Diversity without sovereignty reproduces the extractive pattern at larger scale. Why genetic data are collective Standard research ethics treats a genome as personal information. It is personal, but it is also shared. Each person's genome carries information about parents, siblings, children and more distant relatives, and statistical analysis of many genomes yields information about a population as a whole: its history, its relationships to other groups, the frequency of disease variants within it. That population-level information can be produced from a few dozen samples and applies to everyone in the group, including those who declined to take part. Several kinds of group harm follow. Findings about disease susceptibility can stigmatise a community, as the Havasupai feared. Findings about ancestry can conflict with a people's own account of its origins and, in some legal contexts, with claims to land or recognition. Findings about admixture can be misused by governments or others to question who counts as a member of a people, a concern that Kim TallBear examined in Native American DNA, published in 2013. And a public database of genomes from a small population can make it possible to infer information about identifiable families. Maui Hudson and colleagues, writing in Nature Reviews Genetics in 2020, set out Indigenous perspectives on unrestricted access to genomic data. They argued that open-access policies, which assume that the benefits of sharing outweigh the risks, were designed without regard to collective interests, and that Indigenous peoples should be able to govern access to their genomic data through mechanisms such as controlled access with Indigenous representation, benefit-sharing arrangements, and provenance labelling. Earlier warnings: the diversity projects of the 1990s This is not the first round of the argument. In the early 1990s the Human Genome Diversity Project proposed to collect samples from hundreds of populations around the world, many of them Indigenous, before, as its proponents put it, those populations disappeared through intermarriage and assimilation. The proposal was widely denounced by Indigenous organisations, some of which called it the "vampire project". Critics objected to the salvage framing, to the prospect of cell lines from Indigenous people being patented, and to the absence of Indigenous governance. The project never gained full funding in its planned form. In 2005 National Geographic and IBM launched the Genographic Project to trace human migration through DNA. The Indigenous Peoples Council on Biocolonialism, led by Debra Harry of the Pyramid Lake Paiute, campaigned against it, and the United Nations Permanent Forum on Indigenous Issues recommended that it be suspended in 2006. The project proceeded, largely through public participation kits, and closed its public participation in 2019. These episodes established the Indigenous critique of population genomics that later frameworks drew on. They also show that scientific ambition, left to itself, tends to rediscover the same extractive design every decade. Doing it differently: three models Against that background, several initiatives show how Indigenous governance of genomic research can work. The first model is Indigenous-controlled infrastructure. The Native BioData Consortium, founded in 2018 and based on the lands of the Cheyenne River Sioux Tribe in South Dakota, describes itself as the first Indigenous-led biorepository in the United States. It stores biological samples and data on tribal land, under tribal jurisdiction, and researchers who want to use them must work through its governance. In Australia, the National Centre for Indigenous Genomics at the Australian National University, established in 2016, took custody of a collection of roughly seven thousand blood samples gathered from Aboriginal and Torres Strait Islander people by university researchers between the 1960s and the 1990s. The centre is governed by a board with an Indigenous majority, and its programme has involved consulting the communities from which samples came about whether they want them returned, destroyed or used for research under community-agreed terms. In Aotearoa New Zealand, Genomics Aotearoa established the Aotearoa Genomic Data Repository to keep genomic data from taonga species and other sources within the country under governance informed by Māori principles. The second model is community guidelines with teeth. Te Mata Ira, guidelines for genomic research with Māori published by Hudson and colleagues in 2016, set out how Māori concepts of tapu, mana and whakapapa apply to consent, storage, secondary use and benefit. Later guidelines, Te Nohonga Kaitiaki, extended the approach to genomic research on taonga species, the plants and animals of cultural significance to Māori. Such guidelines give ethics committees and researchers a specific standard to apply rather than general exhortation. The third model is individual practice that anticipates the rules. In 2011 a team led by Eske Willerslev published in Science the first Aboriginal Australian genome, sequenced from a lock of hair given to the British anthropologist Alfred Cort Haddon in the early twentieth century by a man in Western Australia. The researchers had already sequenced the genome when they realised that they had not sought permission from anyone who might speak for the man's descendants. Before publication they consulted the Goldfields Land and Sea Council, the representative body for the region, which endorsed the work. The episode is often cited as a lesson in what should have happened first, and later ancient DNA projects in Australia built consultation into the design from the start. It also shows how much depends on the individual judgement of researchers when rules are absent. A worked example: a kidney disease study done differently To see how these decisions fit together, consider a hypothetical but realistic project. A tribal health department has noticed high rates of early-onset kidney disease among its members and asks a university nephrology group for help. The tribe has about 12,000 enrolled members. The researchers propose a genetic study of 800 participants, combining whole-genome sequencing with clinical records, to look for variants that affect risk and to test whether existing polygenic scores work in this population. Under the old model, the university would obtain approval from its own board, recruit through the tribal clinic, store samples in its biobank, deposit genotype data in a national controlled-access repository as its funder requires, and publish. The tribe would be thanked in the acknowledgements. Under a model consistent with the frameworks in this booklet, the sequence runs differently. The tribal council, advised by its research review board, approves the study by resolution after the protocol has been developed jointly. A governance agreement names the tribe as owner of the samples and data and the university as steward. Samples are processed locally where possible, and residual material is stored either at a tribally controlled facility or at the university under an agreement that the tribe can require return or destruction at any time. Individual consent is tiered: participants choose whether their samples may be used only for kidney disease research, for other health research approved by the tribe, or not stored after the study. The ancestry and population history of the tribe are excluded as research questions unless the tribe later decides otherwise. The data plan agreed with the funder deposits summary statistics from the association analysis in a public resource, since these carry little risk of identifying individuals and are needed for the scientific community to check the findings. Individual-level sequence data are held in a tribally governed enclave, and outside researchers may apply for access to a committee on which tribal representatives hold a majority. The tribe's name appears in the paper only if the council agrees; otherwise the population is described by region and language family. Draft manuscripts go to the review board sixty days before submission. The budget includes salaries for two tribal members trained as research coordinators, a data manager employed by the tribe for the life of the grant, honoraria for an Elders' advisory group at rates comparable to expert consultants, and funds for community meetings at which results are presented in plain language before publication. Two tribal health staff are co-investigators, and one is a co-first author on the main paper. If the study identifies a variant with clinical implications, the agreement commits the university to work with the tribal clinic on a screening programme rather than simply publishing the finding. None of this is exotic. Each element has been used in real projects. What makes the example different from standard practice is that at every decision point the tribe, not the university, holds the final word, and that the budget pays for the tribe to exercise that word competently. Ancestors and ancient DNA Ancient DNA research raises the same issues in a sharper form, because the people sampled cannot consent and their descendants' claims are often contested. The case of the Ancient One, known to scientists as Kennewick Man, is the best known. Skeletal remains around nine thousand years old were found on the banks of the Columbia River in Washington State in 1996. Scientists sued to prevent their repatriation to a coalition of tribes, arguing that the remains could not be shown to be related to living Native Americans, and a federal court agreed in 2004. In 2015 a team including Willerslev published a genome showing that the Ancient One was more closely related to modern Native Americans than to any other population, and to the Colville Tribes, who had supplied comparison samples, in particular. In 2016 Congress passed legislation directing the return of the remains, and in February 2017 the tribes reburied him. The case demonstrates that genomic evidence can support Indigenous claims as well as undermine them, but it also shows how much power sat with scientists and courts. In 2021 a group of more than sixty scholars, including Indigenous researchers, published five guidelines for ancient DNA research in Nature. They called for researchers to follow local regulations, to prepare a detailed plan before studies begin, to minimise damage to remains, to ensure data are made available after publication to allow critical re-examination, and to engage with other stakeholders from the beginning of a study. Some Indigenous scholars criticised the guidelines for treating Indigenous communities as stakeholders among others rather than as holders of authority over their ancestors. The disagreement is instructive. It is the same disagreement, between consultation and consent, that runs through this whole field. Open data, controlled access and the NIH rules Much of the practical conflict comes down to where genomic data are deposited. Journals and funders require data sharing, usually through repositories such as the database of Genotypes and Phenotypes (dbGaP) in the United States or the European Genome-phenome Archive. These repositories offer controlled access, in which applicants must be approved by a data access committee, but the committees are run by the funder or the depositing institution, not by the communities whose data they hold. The NIH Data Management and Sharing Policy, which took effect in January 2023, requires funded researchers to plan for sharing scientific data. Recognising tribal sovereignty, the NIH issued supplemental information on responsible management and sharing of American Indian and Alaska Native participant data. It encourages researchers to engage tribes early, to consider tribal laws and data sharing preferences, and to take account of tribal sovereignty in their data management plans. The guidance does not, however, give tribes a formal veto over NIH repository access decisions. The practical result is that researchers negotiating with tribes must sometimes build into their plans an exception from standard sharing, or a tribally governed access mechanism, and justify it to the funder. A researcher designing a genomic study with an Indigenous partner therefore faces a series of decisions, each of which should be made jointly. Will raw sequence data be deposited at all, or held by the community with summary statistics shared? If deposited, under what access conditions, and with whom on the access committee? Will variant frequencies for the population be published, given that they are population-level information? Will the community's name appear in the paper and the database, or a less identifying label? Can the data be used for ancestry or population history research, or only for the health question agreed? How long will samples be kept, and what happens to them when the project ends? Training Indigenous geneticists The most durable change in genomic research is not a policy but a change in who does the science. The Summer internship for INdigenous peoples in Genomics, known as SING, began in the United States in 2011 and has since spread to Canada, Aotearoa New Zealand and Australia. It trains Indigenous students and community members in genomic science while engaging directly with ethics, governance and history. Katrina Claw and colleagues, writing in Nature Communications in 2018, drew on the SING experience to propose a framework for ethical genomic research with Indigenous communities, emphasising understanding tribal history and sovereignty, community engagement, capacity building, and appropriate governance. A generation of Indigenous geneticists, bioinformaticians and bioethicists now works in universities and tribal organisations. They are changing the field in ways no external guideline could, because they bring to the design of studies the knowledge of what their communities need and fear. For non-Indigenous researchers, the implication is straightforward. The best genomic study with an Indigenous community is increasingly one led, or co-led, by Indigenous scientists, and the role of outsiders is to contribute resources and skills to that leadership rather than to seek partners for their own agenda. Hashtags: #DecolonizingMethodologies #IndigenousEpistemologies #CommunityLedScience #IndigenousResearch #IndigenousDataSovereignty #ResearchSovereignty #IndigenousSelfDetermination #CommunityGovernance #CollectiveConsent #FreePriorAndInformedConsent #CAREPrinciples #OCAPPrinciples #TeManaRaraunga #MaiamNayriWingara #FAIRAndCARE #IndigenousGenomics #BiobankGovernance #ControlledDataAccess #TraditionalKnowledge #BenefitSharing #NagoyaProtocol #CaliFund #IndigenousAuthorship #ReciprocalResearch #FutureOfDecolonizedScience

Download the Book (PDF): Introduction A cryo-electron microscopy project fails, or succeeds, long before anyone looks at a reconstruction. By the time a structural biologist is staring at a density map and arguing about whether a side chain is a leucine or an isoleucine, the ceiling on that argument was fixed weeks earlier, in the two or three seconds during which a thin film of buffer was thinned to a few hundred ångströms and dropped into liquid ethane. Everything that happens afterwards — the choice of accelerating voltage, the exposure regime, the classification strategy, the refinement schedule, the model-building software — can only fail to reach that ceiling. None of it can raise it. This is the single idea the book is built around, and it is worth stating bluntly at the outset because the field's own literature tends to obscure it. Methods papers are written about algorithms, because algorithms are publishable and grids are not. Instrument vendors sell microscopes, because microscopes are expensive and glow discharge units are not. A newcomer reading the recent literature could be forgiven for believing that cryo-EM is a computational discipline with a sample-preparation preamble. It is the opposite: a specimen-preparation discipline with a large and increasingly automated computational tail. The tail matters enormously — a badly processed dataset from an excellent grid yields nothing — but it is a tail. The reason has to do with information, and with the fact that biological cryo-EM operates permanently at the edge of what the physics allows. Electrons interact with matter about five orders of magnitude more strongly than X-rays of comparable wavelength, which is why a single unstained macromolecule can be imaged at all. That same strong interaction destroys the molecule as it images it. Every electron that scatters usefully also deposits energy, breaks bonds, generates radicals, and degrades the very structure whose image it is forming. The consequence is a hard budget: something on the order of a few tens of electrons per square ångström, spent across an entire exposure, after which the high-resolution information in the specimen is gone. Within that budget, each individual image of each individual particle is dominated by noise. The structure emerges only by averaging tens of thousands, sometimes millions, of such images. Because the technique lives on averaging, everything that introduces heterogeneity into the population being averaged costs resolution directly. A particle that is partially denatured at the air–water interface is not a noisy copy of the true structure; it is a different structure, and averaging it in makes the result worse. A particle sitting in ice that is 800 Å thick rather than 400 Å contributes an image with more inelastic background and lower contrast. A particle whose orientation is shared with 90 per cent of its neighbours contributes redundant information along one axis and none along another. These are not processing problems. No classifier recovers information the specimen never contained, and no amount of GPU time compensates for a monolayer of denatured protein at the interface of the film. The practical corollary, which runs through every chapter that follows, is that cryo-EM should be understood as a chain of decisions about where to spend a fixed and non-renewable quantity of information. Vitrification decides how much signal exists. Screening decides whether you find out in an afternoon or in three months. Data collection decides how efficiently the available signal is harvested and how much of it is corrupted by motion, beam-induced charging, or off-axis aberrations. Processing decides how much of the harvested signal survives into the average. Model building decides what claims the resulting map can honestly support. At each link, information can be lost and never recovered. At none of them can it be created. That framing has consequences for how the craft should be learned, and this book is organised accordingly. The first chapter deals with the physics of image formation, because a practitioner who does not understand what the contrast transfer function does to their images will make bad choices about defocus, and a practitioner who does not understand radiation damage will make bad choices about dose. The second and third chapters treat vitrification and the air–water interface, which together constitute the dominant failure mode in the field — probably the majority of stalled cryo-EM projects are stalled there. The fourth chapter is about screening, which is the discipline of finding out quickly, and which is where most of the time in a difficult project is actually saved or lost. Chapters five and six deal with instrumentation and automated acquisition. Chapters seven, eight and nine follow the data from movie frames to a validated map. The final chapter concerns the interpretation of that map as an atomic model, and the ways in which model building can quietly claim more than the density supports. The technique has changed fast enough that a book written five years ago would now mislead in its particulars. Direct electron detectors with counting-mode readout, introduced commercially in the early 2010s, produced what Werner Kühlbrandt named the resolution revolution: structures that had been stuck at 8 Å suddenly resolved side chains. That revolution is over, in the sense that its gains have been absorbed into routine practice. What has happened since is subtler and in some ways more important. Achieving 3 Å has become unremarkable for a well-behaved, reasonably large, compositionally homogeneous complex. Two groups independently demonstrated reconstructions of apoferritin beyond 1.25 Å in 2020, resolving individual atoms and even some hydrogen positions. Energy filters and cold field-emission sources have become standard on new high-end installations. Data collection rates have increased roughly tenfold through beam–image shift acquisition, so that a single overnight session can now yield tens of thousands of movies. Machine-learning particle pickers have largely displaced template matching. Methods for handling continuous conformational heterogeneity — what a molecule does rather than what a single frozen state of it looks like — have moved from research curiosity to routine use. None of this has made specimen preparation easier. If anything, the opposite: as the computational and instrumental links in the chain have improved, the specimen has become an ever more conspicuous bottleneck. A decade ago, a mediocre grid and a mediocre detector produced a mediocre map, and it was not always obvious which was at fault. Today, when a 300 kV energy-filtered instrument with a modern detector and a well-tuned processing pipeline yields a 4.5 Å map of a 400 kDa complex, the fault is almost certainly in the ice. A word on scope. This is a book about single-particle analysis: the reconstruction of a three-dimensional density map from many two-dimensional projections of individual, isolated, purified macromolecules or complexes. It is not a book about cryo-electron tomography, helical reconstruction, or microcrystal electron diffraction, though each is mentioned where the contrast illuminates something about single-particle work. Single-particle analysis is where most practitioners start, where most published cryo-EM structures come from, and where the discipline's logic is clearest. The preparation chapters apply broadly; the reconstruction chapters do not. A second word, on honesty. Cryo-EM produces images that look like objects. This is seductive in a way that diffraction data are not: nobody mistakes a diffraction pattern for a picture of a protein, but a sharpened density map at 3.2 Å looks like a protein, and it is easy to forget that it is a statistical estimate with a resolution-dependent confidence that varies substantially across the volume. The field has a real problem with overclaimed resolution, over-sharpened maps, and models built into density that does not support them. Later chapters treat validation not as a box-ticking exercise at the end but as something that should be running throughout, because the quickest way to produce a wrong structure is to believe a map before checking what it can bear. The reader I have in mind is a structural biologist, biochemist, or biophysicist who has some reason to determine a structure and has concluded, correctly, that cryo-EM is the tool. They may have access to a facility, a collaborator's instrument, or a national centre. They probably do not have three years to spend learning by failure. What follows is an attempt to compress the judgement — about what to look at, what to ignore, when to go back to the biochemistry, and what a map is actually telling you — that otherwise takes those three years to acquire. Chapter 1: What the Electron Sees An unstained protein embedded in amorphous ice is almost invisible. Its atoms are carbon, nitrogen, oxygen and sulphur; the surrounding medium is water, made of oxygen and hydrogen. The difference in mean density between the two is small — protein is roughly 1.35 g/cm³ against water's 0.93 g/cm³ in its vitreous form — and in a conventional bright-field image formed at exact focus, that difference produces essentially no contrast at all. The first thing to understand about cryo-EM is that the images it works with exist only because the microscope is deliberately misused. Amplitude, phase, and why defocus is not a mistake When a coherent electron wave passes through a thin specimen, two things can happen to it. Some electrons are scattered to angles large enough that they are stopped by the objective aperture or lost from the imaging system: this removes amplitude from the transmitted wave and produces amplitude contrast. For biological material in ice at 200–300 kV, amplitude contrast is weak — conventionally modelled as contributing around 7 to 10 per cent of the total contrast, a number that appears in every CTF-fitting program as a fixed parameter. Most of the useful interaction is elastic scattering through small angles, which leaves the electron in the beam but shifts the phase of its wave. A specimen that alters phase without altering amplitude is a pure phase object, and a perfect lens images a pure phase object as a uniform grey field. The information is present in the exit wave; the imaging system simply fails to convert it into intensity. The standard workaround is defocus. Defocusing the objective lens introduces an additional, spatially-frequency-dependent phase shift, which converts part of the phase information into amplitude modulations that a detector can record. The cost is that the conversion is not uniform: it works well at some spatial frequencies, not at all at others, and with inverted sign at others still. This is the contrast transfer function, and it is the central nuisance of the technique. The contrast transfer function The CTF describes, for each spatial frequency, how much of the specimen's phase information is transferred into the recorded image and with what sign. Its dominant term is an oscillating sine function whose argument depends on the defocus, the spherical aberration coefficient of the objective lens, and the electron wavelength. Its practical properties are what matter: It oscillates. As spatial frequency increases, the CTF swings between +1 and −1, crossing zero repeatedly. At every zero crossing, information at that spatial frequency is simply absent from the image. Nothing recovers it from that micrograph. Its oscillation rate depends on defocus. A heavily defocused image has closely spaced zeros; a nearly-in-focus image has widely spaced ones. The first zero — the point below which contrast rises smoothly from zero and above which the first inversion begins — moves to higher resolution as the defocus is reduced. Low frequencies are suppressed. Near zero spatial frequency the CTF goes to zero, which is why very large-scale density variations transfer poorly, and why particles are much easier to see at high defocus, where the first CTF maximum sits at a low spatial frequency corresponding to the overall particle envelope. High frequencies are damped. Beyond the oscillation, an envelope function attenuates transfer at high resolution. Its causes include partial spatial and temporal coherence of the source, chromatic aberration, specimen movement during the exposure, detector modulation transfer, and the intrinsic disorder of the specimen. The composite is usually described by a single Gaussian-like B-factor, and it is the practical limit on resolution in most experiments. The resolution of the reconstruction is not limited by the zeros, because different micrographs collected at different defocus values have zeros in different places. Collect a dataset spanning, say, 0.8 to 2.2 µm underfocus, and every spatial frequency is well-transferred in some substantial fraction of the images. Averaging across the dataset, with the CTF corrected per particle, fills the gaps. This is why defocus is deliberately varied during collection rather than held constant, and why a dataset collected entirely at one defocus value is a mistake even if that value is well chosen. CTF correction itself is straightforward in principle. Fit the function to the power spectrum of each micrograph — the Thon rings, the concentric light and dark bands visible in the Fourier transform of an image of amorphous material, are a direct visualisation of the CTF's squared modulus. Then, during reconstruction, multiply each particle's Fourier transform by the fitted CTF and weight accordingly, which recovers the correct sign everywhere and weights each frequency by how well it was transferred. Programs such as CTFFIND4 and its successors do the fitting in seconds per micrograph and are accurate enough that CTF estimation is rarely the limiting factor; the exception is thick or tilted specimens, where defocus varies significantly across the field of view and per-particle rather than per-micrograph estimation becomes necessary. Phase plates, and why they did not take over If the problem is that a phase object produces no contrast, the obvious fix is to shift the phase of the unscattered beam by a quarter wave relative to the scattered beam, converting phase directly into amplitude near focus. This is Zernike's solution in light microscopy, and it was pursued seriously in electron microscopy for two decades. The Volta phase plate, described by Danev and colleagues in 2014, uses a continuous thin carbon film heated in the beam; the unscattered beam creates a local potential that develops a usable phase shift over time. Phase plates work, and they deliver striking contrast improvements for small particles imaged close to focus. They have nonetheless not become standard practice, for reasons worth understanding because they illustrate a recurring pattern. The phase shift drifts during use and must be monitored; the device introduces its own resolution-limiting scattering; and the main benefit — contrast for particles below roughly 150 kDa — turned out to be partly obtainable by other means, notably better detectors, energy filtering, and improved processing that could align particles with less contrast than was previously assumed. Phase plates remain useful for specific hard cases, particularly small membrane proteins and in tomography, but the field's general answer to low contrast has been to record cleaner images rather than to transform the optics. Radiation damage and the dose budget The reason cryo-EM images are noisy is not that the microscope is inefficient. It is that the specimen can only be illuminated a little before it is destroyed. The numbers are worth internalising. An electron passing through a 100 nm layer of vitreous ice has a substantial probability of scattering inelastically, and each inelastic event deposits on the order of 20 eV — enough to break several covalent bonds. Radiolysis of the surrounding water generates hydroxyl radicals and solvated electrons that attack the specimen chemically. At liquid-nitrogen temperature the products are largely immobilised, which is what makes imaging possible at all, but the bonds are still broken. Cooling to liquid helium temperature buys a modest additional factor, not enough to change practice for most laboratories. Empirically, high-resolution information decays first. Reflections corresponding to 3 Å spacings in a two-dimensional crystal fade after a few electrons per square ångström; information at 10 Å survives to several tens. Richard Henderson's analysis of the physics in the mid-1990s established the theoretical limits of the approach, and the experimental exposure-dependence of signal decay has since been characterised carefully — Grant and Grigorieff published a widely used empirical dose-weighting curve derived from rotavirus particles, which underpins the dose weighting implemented in most current motion-correction software. The practical regime for single-particle work is a total exposure of roughly 30 to 60 e⁻/Ų, fractionated across a movie of perhaps 40 frames. The total is chosen so that low-resolution information, which needs more dose to emerge from noise, is adequately captured; the fractionation is what allows the high-resolution information from the earliest frames to be preserved. Dose weighting is the mechanism. Because each frame of the movie corresponds to a known accumulated exposure, and because the resolution-dependent decay of signal with exposure is approximately known, frames can be low-pass filtered progressively: the first frames contribute their full frequency range, later frames contribute only low frequencies. The result is an exposure-weighted sum that contains more high-resolution signal than any single fixed-dose image could. This is one of the genuine free lunches in the field, and it is a direct consequence of the movie-mode readout that direct detectors made possible. Beam-induced motion Movies also solved, or at least mitigated, a problem that had quietly limited resolution for years. When the beam strikes a vitrified specimen, the specimen moves. The motion has several components: bulk drift of the stage and grid, doming of the suspended ice film within a hole as it responds to charging and stress relaxation, and rotational and translational motion of individual particles. The largest and fastest motion occurs at the beginning of the exposure, exactly when the specimen is least damaged and the high-resolution information is most valuable. Before movie-mode detectors, that motion was integrated into every image as an irrecoverable blur. With movies, the frames can be aligned to each other before summing, recovering much of the lost resolution. Whole-frame alignment removes drift; patch-based or per-particle alignment removes the local, non-uniform component. The combination, as implemented in MotionCor2 and its equivalents, was one of the two technical developments that produced the resolution revolution — the other being the detectors themselves. Motion can also be reduced rather than corrected. Gold foil supports, introduced by Russo and Passmore, deform far less than amorphous carbon under the beam because gold and gold have matched thermal contraction and the material does not accumulate charge in the same way; their all-gold grids substantially reduced beam-induced movement. Thinner ice moves less. Smaller holes move less than larger ones. Reducing motion at source is always better than correcting it afterwards, because correction is itself a noisy estimate. Inelastic scattering, ice thickness, and energy filtering Elastically scattered electrons carry the structural information. Inelastically scattered electrons have lost energy, are chromatically blurred by the objective lens, and contribute a diffuse background that adds noise without adding signal. The fraction of inelastic scattering rises with specimen thickness: the mean free path for inelastic scattering in vitreous ice at 300 kV is on the order of 300 nm, so a specimen 100 nm thick already loses a substantial fraction of its electrons to inelastic events. Two responses follow. The first is to make the ice as thin as the particle allows — a theme that recurs throughout the next two chapters, and the single most consequential variable in specimen preparation. The second is to remove the inelastic electrons optically, using an energy filter that admits only those electrons within a narrow window (typically 10 to 20 eV) around the zero-loss peak. Energy filtering improves the signal-to-noise ratio measurably, with the benefit increasing with thickness. On modern instruments the filter is integrated with the detector and is essentially always on; the practical question is no longer whether to filter but whether one's institution's older instrument has one. Why electrons, and not something else It is worth pausing on the comparison, because it explains why cryo-EM exists in the form it does and why its constraints are not negotiable. X-rays interact with matter weakly. A single protein molecule illuminated by X-rays scatters far too little to form an image; crystallography solves this by using an ordered array of 10¹⁵ copies, so that the scattering adds coherently into diffraction spots. The price is the crystal, which many of the most interesting assemblies will not form. X-ray free-electron lasers offer an alternative — a pulse so short that the diffraction pattern is recorded before the molecule disintegrates — but the facilities are few and the experiments are difficult. Neutrons interact still more weakly, and while their sensitivity to hydrogen makes them uniquely informative for certain questions, the flux available from any existing source rules out imaging single molecules. Electrons sit at the useful point. Their scattering cross-section for the light elements of biology is roughly 10⁵ times that of X-rays at comparable wavelength, which is what makes single-molecule imaging feasible. Crucially, the ratio of useful elastic scattering to damaging energy deposition is more favourable for electrons than for X-rays: per unit of damage inflicted, an electron delivers considerably more structural information. Henderson's 1995 analysis quantified this and concluded that the technique should in principle be capable of atomic resolution from a few tens of thousands of particles — a prediction that looked optimistic for two decades and was finally borne out. The practical corollary is that the dose budget is not a limitation to be engineered away by better instruments. It is a property of the interaction between electrons and organic matter. Every improvement the field has made has consisted of extracting more information from the same fixed number of electrons: better detectors capture more of the electrons that arrive, energy filters discard the ones that carry no information, motion correction stops the specimen from blurring while they arrive, and better algorithms waste fewer of the ones that were recorded. None of it raises the budget. Detective quantum efficiency Everything above concerns what reaches the detector. What the detector does with it is quantified by detective quantum efficiency: the ratio of the squared signal-to-noise ratio at the output to that at the input, as a function of spatial frequency. A perfect detector has DQE of 1 everywhere. Photographic film, which the field used for decades, managed perhaps 0.3 at low frequency and fell off steeply. Scintillator-coupled CCD cameras were worse at high frequency, because the conversion to light spread each electron's signal over several pixels. Direct electron detectors, in which the incident electron generates charge directly in a thin monolithic active pixel sensor, changed this. Their real advantage is not merely higher DQE but the ability to read out fast enough — hundreds of frames per second — that individual electron arrival events can be localised. In counting mode, the detector identifies each electron strike, discards the analogue magnitude of its deposited charge, and records a single count at the estimated position. This removes Landau noise, the variability in energy deposition between individual electrons, which had been a significant noise source. Sub-pixel localisation of the centroid of each event, known as super-resolution counting, can push usable information past the physical Nyquist limit of the sensor. The practical consequences shape modern data collection. Counting requires low dose rate per pixel so that two electrons rarely land in the same pixel in the same frame; coincidence losses otherwise cause non-linearity. That constrains the illumination and therefore the exposure time. The very high frame rates generate enormous data volumes, which the electron-event representation format addresses by storing the list of detected events rather than the images themselves, reducing storage by roughly an order of magnitude while retaining full temporal and sub-pixel information. What this chapter implies for practice The physics sets three constraints that everything downstream must respect. There is a fixed dose budget, so the experiment must extract the most information per electron. Contrast requires defocus, so the dataset must span a range of defocus values and the CTF must be estimated accurately for every particle. Inelastic scattering and beam-induced motion both scale with the thickness and mechanical compliance of the specimen, so thin, well-supported ice is worth more than any instrumental upgrade. That last point is where the next chapter begins. Making ice thin enough to image and cold enough to stay amorphous, without destroying the molecule in the process, is the part of cryo-EM that has resisted automation longest and still decides most projects. Chapter 2: Making Ice That Is Not Ice Water is an inconvenient solvent to freeze. Left to itself it crystallises, and crystalline ice is useless for electron microscopy in three separate ways: the lattice diffracts strongly and swamps the weak signal from the specimen; the growing crystals exclude solute, concentrating salts and buffer components and mechanically disrupting whatever was dissolved in them; and the density changes on crystallisation distort the specimen. The founding insight of the technique, worked out by Jacques Dubochet and his colleagues at EMBL Heidelberg in the early 1980s, was that water can be persuaded to solidify without crystallising if it is cooled fast enough — into a glass, an amorphous solid with the disordered structure of a liquid and the mechanical properties of a solid. Their 1988 review in Quarterly Reviews of Biophysics remains the clearest statement of the principles, and the Nobel Prize in Chemistry in 2017 recognised the work as foundational. The physics of vitrification To vitrify, pure water must be cooled through the region between roughly 230 K and 140 K faster than crystal nuclei can form and grow. The required rate is steep — estimates cluster around 10⁵ to 10⁶ K per second for pure water — and it can only be achieved in very small volumes, because heat has to get out through the surface and the interior of anything thick will cool too slowly. This is why cryo-EM specimens are thin films rather than droplets. A film a few hundred ångströms thick, plunged into a cryogen, vitrifies throughout. A droplet a millimetre across does not: its surface vitrifies and its interior crystallises. The thickness limit for vitrification in pure water is on the order of a micrometre under ideal plunging conditions, which is comfortably above the tens of nanometres that imaging requires, so in practice the imaging constraint binds first — but the two constraints point the same way, and both reward thin films. Solutes help. Glycerol, sugars, high salt and cryoprotectants in general suppress crystal nucleation and lower the required cooling rate, which is why cryo-crystallography can vitrify much larger volumes. Cryo-EM mostly cannot use them: glycerol at the concentrations that would help adds substantially to the background scattering and reduces contrast, and the specimen has to be imaged, not merely preserved. Most cryo-EM buffers are therefore austere — a simple buffer, physiological or near-physiological salt, minimal or no glycerol, and whatever detergent or lipid the specimen genuinely requires. Removing glycerol from a purification that has used it throughout, by a final size-exclusion step into a clean buffer, is a routine and frequently decisive preparatory step. The cryogen Liquid nitrogen is the obvious cryogen and the wrong one. At atmospheric pressure it boils at 77 K, and a room-temperature object plunged into it enters film boiling: a vapour blanket forms around the object and insulates it, so heat transfer collapses and cooling is slow. The Leidenfrost effect that makes water droplets skitter across a hot pan is the same phenomenon in reverse. Liquid ethane, cooled to near its melting point at about 90 K by a surrounding bath of liquid nitrogen, does not boil on contact because its boiling point is far above its melting point. Heat transfer is efficient and the cooling rate is adequate. Propane works similarly; an ethane–propane mixture stays liquid over a wider temperature range and is more forgiving of a cryogen cup that has drifted cold, which is why some laboratories prefer it. The two failure modes at the cryogen are both simple. Ethane too cold becomes slushy and then solid, and a grid plunged into slush does not cool properly and may be physically damaged. Ethane too warm cools too slowly and produces crystalline ice. Modern plungers regulate this reasonably well, but the ethane cup is still a thing to watch, particularly on the first and last grids of a session. After plunging, the grid must never warm above about 140 K until it has been imaged and discarded. Above that temperature vitreous ice devitrifies into cubic ice, and the specimen is lost. Every transfer — into the grid box, into the storage dewar, into the microscope's autoloader cassette — is an opportunity to fail, and a surprising fraction of apparently mysterious bad grids are transfer accidents rather than preparation failures. Grids and support films The specimen sits on a metal grid, typically 3.05 mm in diameter, carrying a perforated support film. The molecules of interest are imaged in the holes, suspended in a free-standing film of vitrified buffer; the support exists to hold that film. The choices here matter more than their cost suggests. Amorphous carbon on copper is the traditional and cheapest combination. Copper conducts heat and electricity well, but copper and carbon have different thermal contraction coefficients, so the composite buckles as it cools and again as the beam heats it, which drives beam-induced motion. Gold foil on gold mesh, developed by Christopher Russo and Lori Passmore, removes that mismatch entirely and reduces beam-induced motion by a large factor; grids of this design have become the default for demanding projects despite costing several times more. Molybdenum sits between the two on contraction matching and is occasionally used for the same reason. The hole geometry is a second axis of choice. Regular arrays of circular holes — 1.2 µm holes on a 1.3 µm pitch is the workhorse, with 2/1, 2/2 and 0.6/1 variants all in common use — give predictable ice gradients and are easy for automated software to target. Lacey carbon, an irregular web, gives a wide range of ice thicknesses in a single square and is useful for screening unknown specimens, but is hard to target automatically. Table 1. Common grid and support combinations, and what each is for. Support Typical use Advantage Caveat Amorphous carbon on copper Routine screening, abundant specimens Cheapest; widely available Thermal mismatch drives beam-induced motion; carbon charges Gold foil on gold mesh High-resolution targets, small particles Minimal beam-induced motion; no charging Cost; more hydrophilic surface may alter particle behaviour Lacey carbon First-pass screening of unknown behaviour Wide range of ice thickness in one square Irregular holes are poor targets for automation Thin continuous carbon over holes Very dilute or interface-sensitive specimens Concentrates particles; may shield from interface Adds background; may induce preferred orientation Graphene or graphene oxide Interface-sensitive, low-concentration specimens Near-transparent; particle adsorption controllable Preparation and quality control are demanding Continuous thin films deserve separate comment because they are the main intervention available when the free-standing ice film is hostile to the specimen. A layer of thin amorphous carbon, graphene oxide, or single-layer graphene stretched across the holes gives particles a surface to adsorb to before they reach the air–water interface. The particle concentration required drops by an order of magnitude or more, which matters for precious specimens. The costs are added background scattering — negligible for graphene, appreciable for carbon — and the risk that adsorption to the film is itself orientation-selective. Graphene in particular has moved from an exotic option to a practical one over the past several years, though preparing reliably clean, uniformly single-layer graphene grids remains a skill that some laboratories have and others do not. Glow discharge and surface chemistry Carbon and gold surfaces as manufactured are hydrophobic; an aqueous droplet beads up rather than spreading. Glow discharge in a low-pressure plasma — air, or air with a controlled admixture of amylamine or oxygen — makes the surface hydrophilic by depositing charged species. The result is a surface that a thin aqueous film will wet. The variables are the atmosphere, the current, and the duration, and they are less standardised across laboratories than they should be. The important practical fact is that the effect decays: a grid glow discharged and left on the bench for an hour behaves differently from one used immediately. Most protocols specify use within a few minutes. The second practical fact is that changing the glow discharge conditions changes particle behaviour on the grid, sometimes dramatically, because it changes the surface charge that the particles encounter. When a specimen distributes badly, glow discharge conditions are among the cheapest variables to explore. Amylamine glow discharge produces a positively charged surface, which can help with specimens that are repelled by the standard negatively charged one. Chemical functionalisation of support films — with affinity tags, lipid monolayers, or antibodies against the target — extends the same logic further and is occasionally the thing that rescues an intractable specimen, though each added layer brings its own background and its own orientation bias. Blotting: the part that is still an art The classical plunge-freezing workflow applies 3 to 4 µl of sample to a glow-discharged grid, presses filter paper against it for a second or two to remove most of the liquid, and immediately plunges the grid into ethane. Commercial plungers — the Vitrobot, Leica's EM GP, the Cryoplunge — control humidity, temperature, blot force, blot time, and the delay before plunging. What they do not control is the physics of what happens in the film. Blotting removes something like 99.9 per cent of the applied liquid in a second. The residue is a film that thins by capillary drainage and evaporation, and its final thickness varies systematically across a grid square and across the grid: thinner near the edge of holes, thicker at the centre, thinner in some regions of the grid than others, in ways that depend on where the filter paper touched, how the grid was held, and local variations in the carbon. This gradient is not a defect to be eliminated. It is the reason a grid usually has some usable area even when the average is wrong, and screening is largely the business of finding where on a given grid the thickness is right. The variables that a practitioner actually controls, in rough order of leverage: sample concentration; blot time; blot force; wait time after application; whether the grid is blotted from one side or both; humidity; and the choice to apply sample twice, blotting between applications, to increase the number of particles in a thin film. Each has to be explored empirically, and the honest description of grid preparation is that one prepares a matrix — typically two or three concentrations against two or three blot conditions — freezes all of them in a single session, and screens them together. There is a strong temptation to treat blotting parameters as a fine-grained continuous optimisation. Resist it. Blot time differences of 0.5 s are within the run-to-run variability of most instruments, and a matrix that varies blot time by 1 s steps across a twofold concentration range covers the useful space more efficiently than one that varies it by 0.2 s at a single concentration. A worked case: membrane proteins The general principles above acquire specific form for membrane proteins, which now constitute a large fraction of cryo-EM targets and which illustrate how the constraints interact. A membrane protein must be kept in a lipid-like environment: a detergent micelle, an amphipol, a lipid nanodisc, or a styrene-maleic acid lipid particle. Each surrounds the transmembrane region with a belt of material that is invisible in the sense of being disordered but very visible in the sense of scattering electrons. This has three consequences. First, the particle is effectively larger than the protein, so the ice must be thicker than the protein alone would require. A 100 kDa transporter in a nanodisc may present a 150 Å object, demanding ice of 200 Å or more. Second, the belt contributes strong low-resolution density that dominates alignment and complicates masking. Refinement can end up aligning on the micelle rather than on the protein, which limits resolution in a way that looks like specimen heterogeneity. Signal subtraction of the belt, or non-uniform refinement with its spatially varying regularisation, addresses this directly and is often worth an ångström or more. Third, free detergent above the critical micelle concentration populates the film with empty micelles, lowering contrast and giving picking algorithms an abundance of plausible-looking objects that are not the target. Keeping the detergent concentration at or just below the CMC after purification, or moving into a nanodisc or amphipol where free detergent is absent, changes the appearance of the grid dramatically. The choice among the reconstitution systems is a real one. Detergent micelles are simplest and most permissive of concentration, but the least native and the most likely to destabilise. Nanodiscs impose a defined lipid environment and a well-behaved, roughly discoid belt, at the cost of a reconstitution step with its own optimisation and a size limit set by the scaffold protein. Amphipols are stabilising and small but give a less defined boundary. Native lipid particles retain the endogenous lipids, which is scientifically attractive and practically demanding. Many projects try two or three; the one that behaves best on grids is not reliably the one that looked best in solution. Blot-free and self-wicking methods Filter-paper blotting has two problems beyond its variability. It takes seconds, during which the sample sits as a thin film exposed to air — enough time, as the next chapter details, for many proteins to find and denature at the air–water interface. And it wastes almost all of the sample. Several systems now avoid it. Dispensing devices such as Spotiton and its commercial descendant Chameleon apply picolitre-scale droplets from a piezoelectric dispenser onto a self-wicking grid — typically nanowire-coated copper — that absorbs excess liquid by capillary action rather than by contact blotting. The interval between application and vitrification can be reduced to tens or even single-digit milliseconds. Sample consumption falls to nanolitres. Other approaches spray the sample onto a moving grid, or use a controlled jet in flight. The shared logic is the same: shorten the time the sample spends as a thin film, and remove the contact step whose reproducibility was always doubtful. For specimens that are stable at the interface these methods offer convenience; for those that are not, they can be the difference between a structure and a project that never worked. They also enable time-resolved experiments, mixing two components milliseconds before vitrification to capture transient states — an application that plunge-freezing by blotting cannot reach. Judging the ice Vitreous ice, examined at low magnification in the microscope, is featureless and grey. Crystalline ice shows as sharp reflections in the diffraction pattern and as textured or granular appearance in the image; hexagonal ice contamination from the atmosphere shows as discrete crystals on the surface. Cubic ice, the product of mild devitrification, is subtler and shows as faint mottling with characteristic diffraction. Thickness is judged by contrast and, more quantitatively, by the ratio of zero-loss to unfiltered intensity on an energy-filtered instrument, or by the appearance of the particles: in ice of the right thickness, particles are visible as distinct grey objects with clear outlines at moderate defocus. In ice that is too thick, they are washed out and the background is grainy. In ice that is too thin, they may be absent entirely — excluded from the thinnest regions — or visibly distorted. The target is ice only slightly thicker than the particle. A 100 Å particle wants perhaps 200 to 400 Å of ice. That sounds absurdly precise, and it is not achieved by control; it is achieved by finding the regions of a variable grid where it happens to hold. Which is the subject of Chapter 4. Before that, though, comes the problem that makes thin ice dangerous as well as desirable. A film 300 Å thick has two surfaces, and a macromolecule diffusing in that film will hit one of them within milliseconds. What happens when it does is the single most consequential unsolved problem in specimen preparation. Chapter 3: The Air–Water Interface Problem Consider the numbers. A vitrified film 500 Å thick has two air–water interfaces separated by 500 Å. A globular protein of 300 kDa diffuses in water with a coefficient on the order of 5 × 10⁻⁷ cm²/s, which corresponds to a root-mean-square displacement of several hundred ångströms in a millisecond. The interval between the end of blotting and vitrification in a conventional plunge-freezer is on the order of a second, and even the fastest dispensing devices operate in the tens of milliseconds. The conclusion is unavoidable: in a conventional preparation, essentially every particle in the film encounters an air–water interface, many times, before the ice forms. The only question is what happens when it does. What the tomography showed For a long time this was a theoretical worry. It became a measured fact largely through cryo-electron tomography of single-particle grids. Alex Noble and colleagues at the New York Structural Biology Center published a systematic tomographic survey of routinely prepared grids in 2018 and found that for the great majority of specimens examined, particles were not distributed through the depth of the ice at all. They were adsorbed to one or both interfaces, often in a monolayer, frequently with a preferred face against the surface. This finding reframed a great deal of accumulated folklore. Preferred orientation, long treated as an occasional nuisance, turned out to be the expected consequence of interface adsorption: a particle that binds the interface through a particular hydrophobic patch will present the same view to the beam every time. Ice that is "too thin" damaging the specimen turned out to be, in part, ice in which both interfaces are close enough that a particle touches both. Specimens that "do not behave on grids" turned out, in many cases, to be specimens that adsorb and unfold. Why the interface is hostile An air–water interface is one of the most denaturing environments a soluble protein can meet. Water molecules at the surface cannot hydrogen-bond upwards; the interfacial tension of pure water at room temperature is about 72 mN/m, and the energetic gradient over the last few ångströms is enormous. A folded protein has a hydrophobic core that is thermodynamically better accommodated at that interface than in bulk water. Adsorption is therefore favourable, and once adsorbed, partial unfolding to expose the core to air is often favourable too. The kinetics matter as much as the thermodynamics. Surface denaturation is not instantaneous — many proteins survive brief interface contact intact — but it proceeds on timescales from milliseconds to seconds, which is precisely the window in which a cryo-EM specimen sits as a thin film. Robert Glaeser and Bong-Gyoon Han set out this analysis clearly, and subsequent work by Edoardo D'Imprima and colleagues provided a vivid demonstration with the tobacco mosaic virus coat protein and with a protein that disassembled into a film of denatured material at the interface, leaving only a minority population intact. The visible consequences on a grid are a characteristic set: · Particles present in the bulk solution but absent or sparse in the ice, because they have adsorbed and aggregated at the interface or, in extreme cases, formed a continuous denatured film. · Particles in overwhelming excess in one orientation, with a corresponding hole in the Fourier space coverage of the reconstruction. · Two-dimensional class averages that look plausible but yield a reconstruction with a smeared or elongated appearance along one axis. · Reconstructions that stall at 6 to 8 Å regardless of how many particles are added, because a substantial fraction of the population is subtly damaged. · A visible grainy or fibrous layer in tomograms of the film, which is the denatured protein. Preferred orientation and its cost Three-dimensional reconstruction from projections requires that the projection directions sample the sphere adequately. If they do not — if a large fraction of particles present the same view — the Fourier transform of the reconstruction has a region of missing or poorly sampled data, and the real-space map is anisotropically blurred. Naydenova and Russo formalised this with an efficiency measure that quantifies how evenly the orientation distribution samples the sphere, giving a single number that predicts how much a given distribution degrades the reconstruction relative to uniform sampling. The practical rule of thumb that preceded it still holds: a distribution with a substantial population in at least three widely separated orientation clusters usually reconstructs adequately; a distribution concentrated in one cluster does not, no matter how many particles it contains. The insidious feature of anisotropy is that global resolution estimates conceal it. A Fourier shell correlation curve averages over all directions, so a map that is 3 Å in two directions and 8 Å in the third can report 3.5 Å. Directional FSC analysis — the 3DFSC approach of Tan and colleagues is the widely used implementation — computes resolution as a function of direction and makes the problem visible. Any structure derived from a dataset with a known orientation bias should report it. Interventions There is no general solution, and anyone claiming one is selling something. There is instead a menu of interventions, each of which changes the physics in a specific way, with specific costs. The practical approach is to work down the menu in roughly increasing order of effort, screening after each change. Table 2. Interventions against interface damage and preferred orientation. Intervention What it changes Cost or risk Detergent below CMC Competes for the interface; occupies surface before protein reaches it Can strip lipids or destabilise complex; narrow effective window Faster vitrification (self-wicking dispensers) Reduces time available for adsorption and unfolding Instrument access; not all specimens benefit Continuous support film (carbon, graphene) Particles adsorb to film rather than interface Added background; film adsorption may itself select orientation Stage tilt during collection Adds projection directions without changing the specimen Thicker apparent ice; defocus gradient; more particles needed Chemical crosslinking (GraFix or in solution) Stabilises the complex against unfolding Can distort structure or fix a non-native state Buffer and additive screening Alters surface activity and particle–interface affinity Empirical; large search space Affinity or functionalised support Binds particles in controlled orientation away from interface Preparation complexity; may restrict orientations further Detergents and surfactants are the first thing most laboratories try, because they are cheap and quick. A small amount of a mild non-ionic detergent — CHAPSO, octyl glucoside, fluorinated surfactants such as FOM or FC-8 — added just before freezing at a concentration below the critical micelle concentration can occupy the interface preferentially and shield the specimen. CHAPSO in particular has become a common first attempt and has rescued a number of published structures. The risks are real: detergents can dissociate complexes, strip essential lipids from membrane proteins, and sometimes thin the ice so aggressively that particles are excluded. The effective concentration window is often narrow and specimen-specific, so this is a screen, not a recipe. Support films move the adsorption event from an air–water interface to a solid–water interface, which is generally gentler. Graphene and graphene oxide are the current preference where available because they add almost nothing to the background. Thin carbon remains widely used. The important caveat: adsorption to a film is still adsorption, and a film that binds one face of the particle preferentially reproduces the orientation problem in a different guise. Functionalising the film — with streptavidin monolayers for biotinylated targets, with nickel-NTA lipid layers for His-tagged constructs, or with immobilised antibodies — gives more control over which face binds, at the cost of substantial preparation effort and an additional layer to see through. Tilting the stage is the intervention that addresses the symptom rather than the cause, and it is often the most reliable. Collecting a proportion of the data at 30 or 40 degrees of tilt converts a single preferred view into a cone of views, filling the missing region of Fourier space. Tan and colleagues demonstrated this systematically in 2017 and it is now a standard rescue. The costs are that the effective ice thickness increases with the secant of the tilt angle, that defocus varies across the field of view and must be handled per-particle, that beam-induced motion is worse, and that more particles are needed because per-particle quality falls. Many practitioners collect a mixed dataset — most micrographs untilted, a substantial fraction at tilt — and combine them, which preserves the high-quality untilted images while supplying the missing directions. Crosslinking with glutaraldehyde, either in solution at low concentration or through a gradient-fixation method such as GraFix, stabilises labile complexes against interface-induced dissociation. It is a blunt instrument. Crosslinking can trap a non-physiological state, can distort surface loops, and can broaden the conformational distribution if it is inhomogeneous. It is nonetheless sometimes the only way to get a fragile assembly into ice intact, and many important structures of large transient complexes were determined from crosslinked material. Buffer optimisation deserves more attention than it usually receives. Ionic strength, pH, the presence of specific ligands or nucleotides, and the concentration of the protein itself all change interfacial behaviour. So does the age of the preparation: a sample that behaves well two hours after elution from a size-exclusion column may behave badly the next morning. The single most common mistake in a stalled cryo-EM project is treating the biochemistry as fixed and the grid conditions as the only variable. Concentration is not a free parameter A recurring novice error is to raise the protein concentration until particles appear in adequate numbers. Sometimes this works. Often it works in a way that is worse than failure, because the excess particles are aggregating at the interface and the apparently dense field contains a mixture of intact and damaged material. The correct reasoning runs the other way. The concentration required to give a good particle density in a film of a given thickness follows from geometry: for a 300 kDa particle in ice a few hundred ångströms thick, something in the range of 0.5 to 2 mg/ml typically gives a workable density if the particles distribute through the film. If a much higher concentration is needed to see anything, particles are being lost — to the interface, to the support film, to the blotting paper, or to aggregation — and the right response is to find out where, not to add more protein. Conversely, if particles are so dense that they overlap and cannot be picked cleanly, diluting is almost always better than trying to pick from a crowded field, because overlapping particles contaminate the alignment of their neighbours. What a successful rescue looks like Abstract menus are less useful than a concrete sequence, so here is the shape of an intervention that works, drawn from the pattern common to many published rescues. A complex of around 250 kDa gives good size-exclusion behaviour and clean two-dimensional classes in negative stain. On cryo grids it appears at adequate density but every class average shows the same top view; the ab initio reconstruction from 60,000 particles is a recognisable but vertically smeared object, and adding particles does not change it. Directional FSC shows resolution of 3.4 Å in the plane and better than 9 Å along the beam direction. The first response is diagnostic rather than corrective: a handful of tomograms confirm that the particles form a monolayer on one surface of the film, all with the same face down. That rules out the possibility that the problem is a processing artefact and identifies which intervention class is relevant. A detergent screen follows — four surfactants at two concentrations each, frozen in a single session. Two of the eight conditions produce no particles at all; one thins the ice so severely that only the hole edges retain material; one produces a visibly broader range of views in the 2D classes, and the corresponding ab initio model is noticeably less anisotropic. That condition is carried forward. It is not, however, sufficient on its own: the orientation distribution has broadened from one cluster to two, which is better but still leaves a gap. A tilted dataset is therefore collected on the improved grids — two-thirds of the micrographs untilted, one-third at 35 degrees — and processed with the tilted micrographs in their own optics groups and with per-particle defocus refinement. The combined reconstruction is isotropic, and the final map reaches 3.0 Å from a particle set only modestly larger than the original. Two features of this sequence are worth extracting. The interventions were combined rather than chosen: the detergent alone and the tilt alone would each have given a partial improvement, and it was their combination that solved the problem. And the diagnostic step came first, which is what made the choice of interventions rational rather than exhaustive. Diagnosing rather than guessing The interventions above are a menu, and working through a menu blindly is slow. Two diagnostic tools shorten the process considerably. The first is tomography of the grid itself. A handful of tomograms of a screening grid — a tilt series through a hole, reconstructed quickly at low resolution — shows directly where the particles are in the depth of the film, whether they form a monolayer at one or both surfaces, and whether a denatured layer is present. This takes an hour of microscope time and can replace weeks of blind condition screening. It is now a standard step in well-run facilities and should be in every practitioner's repertoire. The second is early two-dimensional classification. A few hundred micrographs from a screening session, picked crudely and classified in two dimensions, reveal orientation bias, damage, and compositional heterogeneity within an hour of processing. Classes that all show the same view indicate an orientation problem; classes with fuzzy or missing subunits indicate dissociation; classes that look like fragments or amorphous blobs indicate denaturation. This diagnosis, performed the same day as the screening session rather than a month later, is the single highest-leverage practice in the field. The counsel of realism It is worth saying plainly that this problem has not been solved and that it is not the practitioner's fault when it defeats them. A collective assessment published by several of the field's leading laboratories in 2019 reported, candidly, that even with optimised standard preparation the outcome remains unreliable across specimens — that the same protocol applied to different proteins gives results ranging from excellent to useless, and that nobody can predict which in advance. That admission is useful in two ways. It calibrates expectations: a practitioner whose first six conditions all fail is having a normal experience, not an unusually incompetent one. And it identifies where effort should go. Because the outcome is unpredictable, the efficient strategy is not to reason carefully towards the single best condition but to test many conditions cheaply and quickly. The skill being exercised is experimental throughput, not insight. The corollary is that a project should be organised so that failure is cheap. Freeze grids in batches of eight or twelve rather than two. Keep a stock of several support types in the freezer so that switching costs nothing. Maintain the ability to process a screening dataset the same day. Log every condition and every outcome, so that the pattern across thirty grids is available even though no single grid was informative. Which is the theme of the chapter that follows: how to find out fast. Hashtags: #CryoElectronMicroscopy #CryoEM #SingleParticleAnalysis #SpecimenPreparation #Vitrification #VitreousIce #AirWaterInterface #CryoEMGrids #GlowDischarge #PlungeFreezing #BeamInducedMotion #RadiationDamage #DoseWeighting #ContrastTransferFunction #CTFEstimation #DirectElectronDetectors #EnergyFiltering #AutomatedDataCollection #ParticlePicking #TwoDimensionalClassification #ThreeDimensionalReconstruction #ConformationalHeterogeneity #MapRefinement #AtomicModelBuilding #FutureOfCryoEM

Download the Book (PDF): Introduction In July 2019, in the village of Bana in western Burkina Faso, a small cohort of genetically modified male Anopheles gambiae mosquitoes was released into the open air. The insects carried no gene drive. They were sterile, short-lived, and incapable of leaving descendants. By any molecular standard the experiment was modest: it tested whether laboratory-reared transgenic males could be marked, released, recaptured, and tracked, and whether they behaved in the field anything like their wild counterparts. The scientific yield was correspondingly modest — estimates of survival, dispersal, and recapture rates, later published and showing that the modified males were less fit and less mobile than wild ones. What the release actually cost was not molecular. It took the better part of a decade of regulatory groundwork, an entirely new national biosafety approval process, thousands of hours of village-level engagement, an independent ethics review, a negotiated agreement with local authorities, and a national debate carried out in the presence of organised opposition. And the arc did not end there: in 2025, Burkina Faso's authorities suspended the project's activities in the country altogether, after a period of political turbulence in which the programme's foreign funding and its association with foreign scientific institutions became the story rather than its entomology. A programme that had done more community engagement than any comparable project in the history of vector control lost its licence to operate anyway. That asymmetry — trivial biology, enormous everything-else — is the subject of this book. The genetic engineering of wild populations has passed a threshold. Homing gene drives that spread a genetic construct through a caged mosquito population in a handful of generations have existed since 2016. Suppression drives targeting the doublesex locus in Anopheles gambiae have crashed cage populations to extinction. Self-limiting transgenic mosquitoes have been released in Grand Cayman, Malaysia, Brazil, Panama, and the Florida Keys, in numbers running into the hundreds of millions. A self-limiting diamondback moth has been released in an open field in New York State. Wolbachia-infected Aedes aegypti — not transgenic, but a population-replacement technology in every functional sense — have been deployed across whole cities and have produced the first rigorous randomised evidence that modifying a vector population reduces human disease. The constraint on this field is no longer the construct. It is the trial. The problem this book addresses Testing a genetically engineered organism outside containment is a genuinely novel methodological problem, and the disciplines that would normally supply the method each turn out to supply only part of it. Clinical trial methodology gives us randomisation, blinding, pre-specified endpoints, and the discipline of the protocol — but it assumes the intervention stays where you put it, and that the unit of exposure is a consenting individual. Neither holds. A released mosquito goes where the wind and its own biology take it. A gene drive, by design, goes further than you released it; that is the entire point. Ecological field experimentation gives us replication, controls, baselines, and the hard-won knowledge that natural systems are noisy at exactly the spatial and temporal scales at which we want to detect effects — but ecology's experimental tradition assumes reversibility. You can stop adding nitrogen to a plot. You cannot recall an allele. Pharmaceutical and pesticide regulation gives us dose-response, tiered testing, and environmental risk assessment — but they were built for chemicals that degrade, for products with a manufacturer and a label, and for exposures that can in principle be withdrawn from the market. Public health programme evaluation gives us cluster-randomised designs and the understanding that vector control interventions spill across cluster boundaries — but the spillover it contemplates is of mosquitoes over one season, not of a self-propagating genetic element over decades. And international environmental law gives us the Cartagena Protocol's advance informed agreement procedure, which was drafted with commodity shipments of genetically modified maize in mind, and which struggles with an organism whose defining property is that it crosses borders by itself. So the methodology for field-testing engineered organisms has had to be assembled, in the last ten years, out of pieces of all of these, under real time pressure, and in advance of the technology it is meant to govern. Some of that assembly has been done well. The World Health Organization's guidance framework for testing genetically modified mosquitoes, the United States National Academies' 2016 report on gene drives, the problem-formulation exercises conducted for suppression drives in Anopheles, and the phased pathway published by the scientific working group convened around African malaria elimination all represent serious, careful work. Some of it has been done badly, or has not been done at all: there is still no agreed answer to the question of who is entitled to consent to a release whose effects cross a national border, and no agreed standard for how long post-release monitoring must continue, or who pays for it when the funder's grant ends. The controlling argument This book makes one central claim, and the chapters develop it: The binding constraint on genetic vector control is the design of the trial, not the design of the construct — and a trial is adequate only when its spatial scale, its monitoring power, its statistical design, its regulatory basis, and its social mandate are all sized to the same organism. That last clause is where most real programmes fail. A trial can be beautifully randomised and statistically powerful, and still useless because the treated and control clusters are three kilometres apart and the vector routinely moves five. It can be perfectly scaled to the vector's dispersal and still illegitimate, because the villages inside the release zone were consulted and the ones just outside it were not. It can be legally impeccable under national law and still a transboundary act, because the river that marks the district boundary does not mark the mosquito's. It can have genuine community agreement and still be uninterpretable, because the national bed-net campaign happened to run during the trial year and nobody accounted for it. The five sizings — spatial, statistical, ecological, legal, and social — are not a checklist to be worked through in series. They constrain each other. Widening the release zone to achieve isolation increases the number of communities whose agreement is needed and the number of traps that must be serviced. Increasing monitoring intensity improves detection limits but changes the behaviour of the thing being monitored, because household-level trapping is itself an intervention in the community's relationship with the project. Choosing an island for its isolation buys clean inference and buys, along with it, a population whose ecology may generalise to nowhere. Getting these to cohere is a design problem, and it is the one this book teaches. What is in scope and what is not This is a book about testing and governance. It covers how to decide what evidence a release must generate; how to formulate the risk questions before generating data; how to measure dispersal and what dispersal measurements can and cannot support; how to model release and spread in a way that informs trial design rather than pretending to forecast; how to choose a site; how to monitor entomologically and molecularly, and how to know what your detection limit really is; how to design an epidemiological trial when the intervention leaks; how the international and national regulatory instruments actually work and where they do not; how communities can meaningfully agree to something that cannot be individually consented to; and what it means to design a programme for a technology some of whose effects cannot be undone. It does not cover the molecular construction of drive systems. There are no protocols here — no guide cassettes, no promoter choices, no methods for improving homing rates or engineering resistance-refractory target sites. That literature exists, it is published, and it is not what limits the field. Anyone whose interest in gene drives is constructional will find this book useless, which is the intent. It is also, deliberately, not an advocacy document in either direction. Malaria killed well over half a million people in 2023, the overwhelming majority of them African children under five, and the tools that drove mortality down in the 2000s — insecticide-treated nets and artemisinin combination therapy — are losing ground to pyrethroid resistance, to Anopheles stephensi invading African cities, and to flat financing. That is a real emergency, and it is the reason serious people are willing to contemplate engineering a wild species. It is also not, by itself, an argument that any particular release is safe or legitimate. Urgency is a reason to test well and quickly. It has never been a reason to test badly. How the argument proceeds The book begins with what is actually being released, because the single most consequential decision in a trial programme — more consequential than any statistical choice — is which category of organism is in the cage, and how far into the future and the landscape it can propagate. From there it moves to the phased pathway that structures the whole enterprise, and to the problem-formulation discipline that determines what a phase is supposed to find out. The middle chapters are the quantitative core: dispersal measurement, population and drive modelling, site selection, entomological and molecular monitoring, and epidemiological trial design. These are the chapters where most of the methodological work sits, and they are written for a reader who wants to know what the models actually assume, where the estimates come from, and how much weight they will bear. The last chapters turn to the institutional and social architecture — regulation, transboundary consequence, liability, and the consent problem — and the conclusion takes up the question that genuinely distinguishes this technology from everything else in the public health toolkit: how to design a programme when part of what you are doing cannot be taken back. A reader who finishes should be able to look at a proposed field trial of an engineered organism and say, with specificity, what it can establish, what it cannot, what would have to change for it to establish more, and whether the people who will live with the result have been given any real say in it. Chapter 1: What Is Actually Being Released Every methodological question in this book has a different answer depending on one prior fact: how far into the future, and how far across the landscape, the released material can propagate itself. A trial of an organism that cannot leave descendants is a logistics exercise with an ecological monitoring component. A trial of an organism carrying a low-threshold homing drive is an intervention in the evolutionary trajectory of a species, and no amount of careful plot design changes that. Collapsing these into one category — "GM mosquitoes" — is the most common and most consequential error in the public debate, and it is made by proponents and opponents alike. So the first task is taxonomy: not of the organisms, but of the propagation behaviour of what they carry. Five categories, not one Self-limiting, non-persisting. The released individuals carry a genetic construct that kills them or their offspring, so the transgene declines geometrically from the moment releases stop. Oxitec's OX513A Aedes aegypti — the first widely deployed example — carried a repressible lethal system such that offspring of released males died before adulthood unless supplied with tetracycline in the rearing facility. Its successor, OX5034, is female-lethal: male offspring survive and carry the construct forward for several generations, so suppression persists longer after each release, but the trajectory is still downward. Target Malaria's first-generation Anopheles releases used a male-sterile line with no inheritance at all. The defining property is that the transgene has a negative fitness effect by construction, so the population genetics are a decay curve; the only question is the half-life. Self-limiting with engineered persistence. Between pure decay and true drive sits a band of constructs designed to persist for a bounded number of generations or below a confinement threshold: daisy-chain arrangements in which drive components are arrayed so that the chain exhausts itself, split-drive systems where the nuclease and the guide are separated so that the drive cannot propagate without both, and killer-rescue and underdominance systems that only establish above a high release frequency. These are, so far, laboratory constructs rather than field candidates, but they matter to trial design because they are the only known technical route to a release that is genuinely spatially confinable. For a methodologist they are the interesting middle: they permit a field experiment whose results extrapolate to a drive without being one. Non-transgenic population modification. Wolbachia-infected Aedes aegypti are not genetically modified in the regulatory sense in most jurisdictions, but functionally they are a population-replacement technology: the wMel strain spreads through a wild population by cytoplasmic incompatibility, establishes at high frequency, and persists indefinitely, reducing the mosquito's competence to transmit dengue. It is worth treating alongside the engineered technologies because it is the only one that has completed a rigorous randomised trial with a human disease endpoint, and because its regulatory treatment — light in most countries, because no recombinant DNA is involved — illustrates how poorly regulatory triggers track actual ecological consequence. Wolbachia replacement is, in ecological terms, a more permanent intervention than anything Oxitec has released, and in most jurisdictions it faced a lower bar. High-threshold or "low-invasiveness" drives. Drive systems whose dynamics include a frequency threshold below which they are eliminated by selection. Released above that threshold locally, they spread to fixation within the release area; migrants carrying the construct into a neighbouring population arrive below the threshold and are lost. This is the property that would, in principle, permit a genuinely local field trial of a self-propagating element. The engineering is not there yet at field scale, but the concept structures a great deal of the regulatory thinking about what a confinable drive trial would look like. Low-threshold homing drives. A homing endonuclease drive cuts the homologous chromosome at its own target site and is copied across during repair, so that a heterozygote transmits the construct to far more than half its offspring. Above a very low introduction frequency, such a construct is expected to spread through a connected population regardless of where it started. Suppression drives of this class targeting the doublesex locus in Anopheles gambiae have driven caged populations to collapse. These are the constructs that make this an entirely different governance problem: the release site is not the exposure site, the trial population is not the affected population, and the experiment has no natural end. Table 1 sets out how the five categories differ on the properties that actually drive trial design. Table 1. Release categories and what each implies for trial design. Category Persistence after release stops Expected spatial spread Practical reversibility Trial unit Self-limiting, non-persisting Weeks to a few generations Release area plus dispersal range Stop releasing Treated area vs control area Self-limiting, engineered persistence Bounded number of generations Local, threshold-constrained Stop releasing; wait out chain Treated area vs control area Wolbachia replacement Indefinite Spreads to connected population None practical City or district High-threshold drive Indefinite where established Local if threshold holds None practical; depends on threshold Isolated population Low-threshold homing drive Indefinite Whole connected species range None demonstrated The species The rightmost column is the one that unsettles people, and it should. For a low-threshold drive, the honest statement of the experimental unit is not the village, the district, or the country. It is the interbreeding population — which for Anopheles gambiae sensu lato is, on current evidence about gene flow, something close to continental. The published field record It helps to be concrete about what has actually been released outdoors, because the debate is frequently conducted as though gene drives had already been in the field. Oxitec's Aedes aegypti programme is the largest body of open-release experience. Small-scale releases on Grand Cayman in 2009–2010 provided the first field data on mating competitiveness of released transgenic males; a trial at Bentong in Malaysia in 2010–2011 produced mark-release-recapture estimates; larger suppression trials followed in Juazeiro and Jacobina in Brazil and in Panama. Brazil's CTNBio approved OX513A for commercial release in 2014, the first such approval anywhere. In the United States, after a protracted and instructive public process in the Florida Keys, the Environmental Protection Agency issued an experimental use permit and OX5034 mosquitoes were released from 2021. Target Malaria's Anopheles programme is the only open release of a genetically modified malaria vector. The 2019 Bana release used sterile males with no drive, and was explicitly framed as the first step of a phased pathway whose later steps would involve, successively, a self-limiting fertile male-bias construct and eventually a suppression drive. The entomological results — reduced survival and shorter dispersal for the modified males than for wild-type controls — were published in 2022 and are a useful reminder that laboratory-reared, marked, transgenic insects are not ecologically equivalent to wild ones, which has direct consequences for how release ratios are calculated. The Wolbachia programme is the largest by area. Deployments have covered whole cities in Indonesia, Vietnam, Brazil, Colombia, and Australia. The Applying Wolbachia to Eliminate Dengue trial in Yogyakarta, Indonesia, published in the New England Journal of Medicine in 2021, was a cluster-randomised trial with a test-negative design across 24 clusters, and reported a 77% reduction in symptomatic dengue incidence in intervention clusters. It remains the single strongest piece of evidence that modifying a vector population changes human disease outcomes, and it is worth studying as a design as much as a result. Agricultural releases are the forgotten precedent. A self-limiting diamondback moth developed by Oxitec and evaluated by Cornell researchers was released in an open field in New York State, with results published in 2020. It attracted a fraction of the attention that a mosquito release does, which tells us something about how much of the governance burden here is attached to the vector and the setting rather than to the technology. No gene drive organism has been released anywhere. Every drive result to date is from cages, from population cages of increasing ecological realism, or from simulation. That gap — between a technology whose cage performance is established and a testing pathway that has not yet reached its first confined field step — is exactly the space this book occupies. Suppression versus modification Cutting across the propagation categories is a second distinction with different methodological consequences: whether the intervention aims to reduce the vector population or to change what it carries. Suppression reduces vector density, ideally below the threshold at which transmission is sustained. Its monitoring endpoint is a count: adults per trap-night, larval indices, human biting rate. Counts of mosquitoes are noisy, seasonal, and strongly non-linear in rainfall, so detecting a suppression effect requires either a large effect, long baselines, or both. Its ecological risk questions are about the consequences of removing a species or a substantial part of its biomass from a food web, and about niche vacancy — whether a suppressed Anopheles gambiae population is replaced by a secondary vector that is harder to control. Modification leaves the population size alone and replaces its genotype with one that cannot transmit the pathogen. Its monitoring endpoint is a frequency: what proportion of sampled mosquitoes carry the construct. Frequencies are estimated from genotyping, which has different and generally better statistical properties than counting — you can get a tight confidence interval on an allele frequency from a few hundred insects. Its ecological risk questions are different too: the population remains, so food-web arguments largely fall away, and attention shifts to the fitness cost of the construct, the evolution of resistance, and the possibility of the modification moving into related species through hybridisation. The two strategies fail in different ways, and a trial designed for one will be badly designed for the other. A suppression trial needs high trap density and a long baseline; a modification trial needs a genotyping pipeline and a sampling design that gives unbiased frequency estimates across age classes and habitats. Programmes that hedge between the two — releasing something that both suppresses and modifies — need both, and typically budget for neither. Why the category decides everything downstream Consider a single design question — how far apart the treated and untreated areas must be — and watch the answer change across the categories. For a self-limiting release, the requirement is that few enough released males reach the control area to contaminate the comparison. Since released males are short-lived, this is a dispersal question with a defined time horizon, and a separation of a few kilometres with some geographic discontinuity is usually defensible. For Wolbachia, the requirement is that the infection does not cross into control clusters during the trial period; the Yogyakarta trial handled this by accepting and modelling contamination rather than pretending it away, and by using a design robust to it. For a low-threshold drive, there is no separation that satisfies the requirement, because the requirement cannot be satisfied by distance. A control area that is connected by any gene flow will eventually become a treated area. The comparison has to come from time, from unconnected populations far outside the range of spread, or from nowhere. The same collapse happens with consent. For a self-limiting release the affected community is identifiable and bounded — imperfectly, but recognisably. For a drive it is not, and the consent question either becomes a question about international governance or it becomes a fiction. And it happens with monitoring. Monitoring a self-limiting release ends when the transgene is undetectable; that is a determinate event that can be written into a protocol and a budget. Monitoring a drive release has no natural endpoint at all, which is a problem for research funding on three-to-five-year cycles and for national agencies whose entomological surveillance capacity is already committed. The organism is not only the construct A release is an event involving a whole insect, reared in a factory, sorted by sex, transported, and let go at a particular hour of a particular day. Several of the properties that determine whether the trial works are properties of that process rather than of the genetics, and they are routinely under-specified in protocols. Rearing history. Mass-reared insects differ systematically from wild ones. They are selected, inadvertently, for tolerance of crowded larval trays, for mating in confined spaces, and for whatever the colony's diet and light cycle reward. Colonies drift; a line maintained for fifty generations is not the line that was characterised at generation five. Good practice is periodic outcrossing to wild material from the release area, and it has a cost: each outcross re-introduces genetic background variation that the characterisation data were not collected on. A trial protocol should state the colony's generation count, its outcrossing schedule, and the wild source population, and should treat these as version information about the thing being released. Sex separation. Almost every mosquito release is male-only, because female mosquitoes bite and released biting females are both an epidemiological and a political problem. Separation methods — pupal size sorting, genetic sexing strains, optical sorting — have error rates, and the error rate matters enormously to community acceptance. A release of ten million males with a female contamination rate of one in ten thousand releases a thousand biting females. The correct handling is to state the measured contamination rate with its confidence interval, to report it per batch rather than as a design figure, and to have agreed in advance with the regulator and the community what rate triggers a halt. Release ratio and competitiveness. Suppression by self-limiting males depends on released males successfully competing for wild females. The standard summary is the mating competitiveness index, estimated from field cage or mark-release-recapture data, and it is almost always well below one. If released males are half as competitive as wild males, achieving an effective ratio of five-to-one requires releasing ten-to-one. The Burkina Faso results — modified sterile males with reduced survival and shorter dispersal than wild males — are exactly the sort of finding that changes a release schedule, and they were only obtainable from a field release. This is the strongest argument for small non-drive field steps in a phased programme: laboratory competitiveness estimates are systematically optimistic, and there is no way to find out by how much except outdoors. Timing and distribution. Where releases happen within a village, how many release points there are, and how they are spaced relative to larval habitat determine the realised spatial distribution of the intervention. A programme that releases from a vehicle along the main road is treating a different area from one that releases at fifty household points, even if the counts are identical. Species biology constrains the design more than the construct does Three vector systems dominate the field literature, and they impose very different trial designs. Aedes aegypti is domestic, container-breeding, and short-ranged. Mark-release-recapture studies consistently find that most individuals stay within a few hundred metres of their release point, often within a single block. That short range is why Aedes trials can use urban clusters of a kilometre or two across and expect only moderate contamination, and why Aedes work has produced the majority of the field record. It also means the intervention is intimately domestic: the mosquitoes being released will breed in the water containers of the households being consulted, which changes the texture of community engagement. Anopheles gambiae sensu lato is a species complex, not a species — it includes An. gambiae sensu stricto, An. coluzzii, An. arabiensis, and others that hybridise at low rates and occupy overlapping but distinct habitats. A construct released into one member of the complex can plausibly introgress into another, which is a risk-assessment question for a self-limiting release and an entirely different question for a drive. The complex is also seasonal to an extreme degree in the Sahel: populations crash in the dry season to numbers so low that they are difficult to sample at all, then rebound. How a population persists through the dry season — aestivation, local refugia, or long-distance re-colonisation — is not fully resolved, and the answer changes both the suppression target and the spatial scale of the trial. Anopheles stephensi is the newest complication. An urban-adapted vector historically confined to South Asia and the Arabian peninsula, it has established in the Horn of Africa and been detected progressively further west, and it breeds in the constructed water storage of African cities in a way An. gambiae does not. A suppression programme that removes An. gambiae from a landscape in which An. stephensi is establishing has not necessarily removed transmission; it may have cleared a niche. Any serious ecological model of a suppression trial now has to contemplate a second vector, and any monitoring protocol has to be able to identify it. The lesson for design is that the construct is portable across species and the trial is not. A drive that works in An. gambiae tells you very little about how a drive trial in Aedes should be laid out, because the two insects live at different spatial scales, in different relationships to human settlement, with different seasonal dynamics and different sibling-species complications. The honest framing for a trial programme The practical implication is that a trial programme should be explicit, in its first document, about which category it is in, and should refuse the rhetorical convenience of borrowing another category's properties. Two moves in particular are illegitimate. The first is borrowing self-limiting reversibility to reassure about a drive: "if anything goes wrong, we stop releasing" is true of Oxitec's product and meaningless for a homing drive. The second is borrowing drive-scale ambition to justify a self-limiting trial: a self-limiting release does not deliver malaria elimination and should not be sold as a step towards it except in the specific and defensible sense that it is a step along a phased pathway whose later steps are different technologies requiring their own approvals. Target Malaria has generally been careful about this, describing its pathway in three distinct technology steps rather than one continuous programme, and seeking separate approvals for each. That care did not save it from having its activities suspended in Burkina Faso in 2025 — which is its own lesson, taken up later in this book — but it is the right practice, and the reason is methodological as much as ethical. Each category generates different evidence, and evidence generated about a sterile male release simply does not transfer to a drive. A pathway that pretends otherwise is not building a case; it is building a false one, and the first serious regulatory review will say so. Chapter 2: The Phased Pathway from Cage to Landscape The organising device of the whole enterprise is the phased pathway: a sequence of testing stages, each in a more open setting than the last, with defined criteria that must be met before the next begins. It is borrowed from clinical development, where phase I, II and III trials serve different purposes and a drug that fails one does not proceed to the next. The World Health Organization's guidance framework for testing genetically modified mosquitoes formalised it for this field, and it now structures almost every programme document, regulatory submission, and funder review in the area. The borrowing is productive but imperfect, and understanding where the analogy breaks is the beginning of using the pathway well. The stages Stage 1: laboratory and contained population studies. Characterisation of the construct and the line: insertion site, copy number, stability over generations, expression, fitness costs measured in cages, mating competitiveness in small arenas, and for drives, the homing rate, the rate of generation of resistance alleles, and the trajectory of caged populations. Containment here is physical and, for organisms of concern, ecological and molecular as well — facilities are sited outside the vector's climatic range, and constructs may be split so that the drive is non-functional as a unit. Stage 2: large cages and physically confined field-like conditions. Field cages of tens of cubic metres, sited in the target country, with natural light, humidity, and swarming space. This stage exists because mating behaviour in Anopheles is a swarm phenomenon that small laboratory cages suppress: a line that mates perfectly well in a bucket may not form or join swarms. Large-cage studies are where fitness and competitiveness estimates first become credible, and where drive dynamics can be observed in a population with realistic age structure. Stage 3: small-scale, staged open release. The first outdoor release, sized to generate data on survival, dispersal, and behaviour rather than to achieve an epidemiological effect. For self-limiting organisms this is a mark-release-recapture study. The Bana release in 2019 was a stage 3 study in every respect: a few thousand sterile males, an intensive recapture effort across the village, and no expectation that the mosquito population would change. Stage 4: larger open release with entomological endpoints. Releases at a scale intended to change vector density or genotype frequency in a defined area, with a comparison area, over at least one full transmission season. This is where suppression or replacement is demonstrated as an entomological fact. Stage 5: epidemiological trial. A trial powered on human disease outcomes — infection incidence, clinical case incidence, parasite prevalence — typically cluster-randomised, and typically the most expensive single element of the programme by an order of magnitude. Stage 6: post-implementation surveillance. Ongoing monitoring after deployment for effects not detectable in a trial: resistance evolution, spread beyond the intended area, ecological changes with long lags, and epidemiological rebound. Table 2 sets out what each stage can and cannot establish. Table 2. Testing stages, their questions, and their limits. Stage Setting Primary question What it cannot establish 1 Laboratory, contained Does the construct behave as designed? Field fitness; ecological interactions 2 Large cages in-country Does it behave in realistic mating conditions? Dispersal; landscape-scale dynamics 3 Small open release Do released insects survive, disperse and behave as modelled? Population-level effect 4 Larger open release Does vector density or genotype frequency change? Effect on human disease 5 Cluster-randomised trial Does it reduce human infection or disease? Long-lag ecological effects; resistance 6 Post-deployment surveillance What emerges over years? Nothing prospectively; it is detection, not test Where the clinical analogy holds Three features transfer well. Stage-gating as a decision discipline. The pathway's real function is not to arrange experiments in order of size — that would happen anyway — but to force a programme to specify, in advance, what result would stop it. In drug development this is embedded in institutional practice: a phase II failure ends the programme. In this field, stage-gate criteria are frequently written in language so soft that no observable result could fail them. "Proceed if no unacceptable adverse effects are observed" is not a criterion; it is a mood. A usable criterion names a measurable quantity, a threshold, and a decision: proceed to stage 4 only if the estimated mean dispersal distance is below X metres with the upper 95% bound below Y; halt if female contamination in any release batch exceeds one in Z. Escalating exposure. Each stage exposes more of the world to the intervention, and the pathway's logic is that you buy information cheaply at low exposure before you pay for it at high exposure. This is precisely the phase I logic of dose escalation, and it is sound. Protocol discipline. Pre-specification of endpoints and analysis, registration of the trial, independent monitoring, and publication of results regardless of direction are all clinical-trial habits this field has adopted with varying rigour and should adopt completely. Where it breaks Exposure does not de-escalate. In a clinical trial, phase I subjects stop taking the drug. In a field release of a persisting organism, the exposure created at stage 3 does not end when stage 3 does. For self-limiting releases this is nearly true and the analogy nearly holds. For a drive it fails entirely: the first open release of a low-threshold drive is simultaneously stage 3, stage 4, stage 5 and stage 6, because there is no smaller version of releasing a self-propagating element. You can release fewer individuals, which changes the establishment probability and the time course, but not the eventual extent. This is the single most important structural fact about the pathway, and it has an uncomfortable implication: for low-threshold drives, the pathway's escalation logic terminates at stage 2 unless a confinement mechanism is available. Everything past that point is either an unconfined release or requires an ecologically isolated population — an island, or a population with no gene flow to the rest of the species. The serious proposals in this area accept this and structure their pathway around it: use self-limiting and non-drive constructs to establish field behaviour, use high-threshold or split systems to test drive dynamics in the open where possible, and treat the first low-threshold release as the terminal decision it is. There is no placebo and no blinding. Vector control cannot be blinded from the community, only from the laboratory technicians who process samples and from the clinicians who diagnose outcomes. That is worth doing — the Yogyakarta trial's use of a test-negative design blinded outcome ascertainment effectively — but the fact that everyone knows which villages were treated introduces reporting and care-seeking differences that must be measured rather than assumed away. The control arm is not untreated. Every malaria trial arm has bed nets, because withholding them is unethical, and increasingly has seasonal chemoprevention and in some settings a vaccine. The intervention is therefore always tested as an addition to a control package that is itself changing during the trial, often because of a national campaign the trial does not control. The effect size being estimated is an increment, and it is an increment over a moving baseline. The sponsor cannot withdraw the product. Pharmacovigilance rests ultimately on the ability to recall a marketed drug. There is no recall for an established construct. Stage 6 surveillance therefore has a different logical status: it can detect, and it can inform mitigation, but it cannot restore the prior state. Programmes that describe post-release monitoring as "a safety net" are using a metaphor that implies catching something before it hits the ground. Mostly it does not. What each stage should actually be sized to detect A recurring weakness in published protocols is that stages are sized by budget and convenience and then described as though the size were chosen for statistical reasons. Doing it properly means working backwards from the decision each stage informs. Stage 3 is sized by the precision needed on dispersal. If the purpose of the small release is to estimate a dispersal kernel that will determine the isolation distance for stage 4, then the release size and the recapture effort should be set so that the upper confidence bound on the relevant distance percentile is tight enough to distinguish the candidate site separations. Mark-release-recapture recovery rates for Anopheles are often well under one percent, which means the number released has to be large enough that a fraction of a percent is still a usable sample, and the trap array has to extend far enough that the tail of the kernel is observable at all. A trap array that stops at 500 metres cannot estimate a tail beyond 500 metres; it can only fail to detect it, which is a different statement and is routinely reported as though it were the first. Stage 4 is sized by the entomological effect and the variance of mosquito counts. Trap counts are overdispersed, seasonal, and spatially autocorrelated. The relevant variance is between-cluster variance in the outcome after accounting for season, not the sampling variance of individual trap nights, and it is usually estimated badly. Long pre-intervention baselines are the most efficient way to buy power here, because they let each cluster serve partly as its own control. Stage 5 is sized by disease incidence, cluster count and contamination. Power in a cluster-randomised trial is dominated by the number of clusters, not the number of people, once clusters are reasonably large. Contamination between arms biases the estimate towards the null. Both facts push towards fewer, larger, more separated clusters, and both collide with the cost of monitoring and the availability of suitable sites. Confinement is five different things The pathway's escalation is usually described as moving from "contained" to "uncontained", which is too coarse. Confinement in this field has at least five independent dimensions, and a given stage may be strong on some and weak on others. Distinguishing them is what allows a programme to design an intermediate step rather than jumping from a cage to a landscape. Physical confinement is the insectary: screens, airlocks, negative pressure, double doors, sealed drains, and the procedural apparatus of gowning and logging. It is mature technology, borrowed directly from arthropod containment practice, and it fails in the ways all physical containment fails — through human error, through infrastructure failure, and through transport. Geographic confinement sites the facility outside the target organism's climatic or ecological range, so that an escape cannot establish. A temperate-country insectary working on a tropical vector has geographic confinement; an insectary in Bobo-Dioulasso working on Anopheles gambiae does not, and that is precisely why in-country facilities carry a much heavier physical containment burden. Ecological confinement relies on the absence of mates, hosts, or larval habitat. It is real but fragile, because it depends on an ecological claim that is rarely tested with the rigour of the engineering claims around it. Reproductive confinement is built into the organism: sterility, sex-limited lethality, or dependence on a dietary supplement such as tetracycline. It is the confinement that self-limiting products rely on, and it degrades gracefully rather than catastrophically — a failure produces some surviving offspring, not an established population. Molecular confinement separates the drive into components that are individually non-functional, so that an escaped insect carries a cassette that cannot propagate. Split drives are the clearest case: the nuclease sits at one locus and the guide at another, and only the laboratory line has both. This is the most important confinement category for drive research, because it is what allows genuinely informative work to proceed at low risk. A programme's containment statement should enumerate all five and say which ones it is relying on. In practice most rely on physical confinement plus one other, and the combinations differ by stage. The value of the enumeration is that it exposes the single points of failure: a project relying on physical confinement alone, in-country, with a fully functional drive and no molecular split, has one barrier between the laboratory and the landscape, however many doors are on it. Data, samples and the obligations that outlive the stage Each stage generates material and records whose handling is part of the methodology and is almost always treated as an afterthought. Specimen archives. Every mosquito collected in a monitoring programme is a permanent record of the genetic state of that population at that date, and an archive of them is the only way to answer questions that are not yet being asked. The Jacobina episode in Brazil illustrates the point: a 2019 paper reported that genetic material from released OX513A mosquitoes was detectable in the local Aedes aegypti population, a finding whose interpretation was heavily contested and which prompted an editorial note on the paper. Whatever one concludes about that specific dispute, it could only be had at all because samples existed from before, during and after the release. A programme that discards its specimens after genotyping for its primary endpoint has destroyed the evidence base for every future question, including the ones its critics will ask. Baseline genomic data. For any release into a population, a pre-release genomic baseline — allele frequencies at the target locus, standing variation in the target sequence, population structure and gene flow estimates — is both a scientific necessity and a defensive one. Resistance-relevant standing variation at a drive target site is the clearest example: without a baseline you cannot distinguish resistance that arose in response to the drive from variation that was always there. Programmes should collect and publish this before any release, and for Anopheles the collaborative genomic resources assembled over the past decade make this far easier than it once was. Data that survives the funder. Stage 6 obligations run for decades; grants run for five years. Where the monitoring data live, who can access them, and what happens to them when the project's institutions change or leave the country are questions that should be answered in writing before stage 3, not after stage 5. The most robust arrangements place data and specimens with the national research institution rather than the international partner, which has the additional merit of aligning the scientific capital with the country bearing the risk. The pathway as a social object There is a second function of the phased pathway that the clinical analogy obscures: it is a device for building institutional and public trust over time, and it fails in that role for reasons that have nothing to do with statistics. The staged approach lets a national regulator gain experience with a novel product class before facing the hardest decision. Burkina Faso's biosafety agency had to build a review capacity that did not previously exist; reviewing a sterile male release is a manageable first exercise, reviewing a drive is not. Similarly, communities acquire experience of what a release actually looks like — vehicles, traps, people in the village, no visible consequence — which is information they cannot get from a consultation document. But the pathway also creates a commitment problem. A programme that has invested a decade and substantial funding in stages 1 to 3 has powerful institutional reasons to proceed to stage 4, and the people reviewing the stage-gate are frequently the people who built the programme. Opponents read the pathway as a ratchet, and they are not wrong to: no major programme in this field has yet voluntarily stopped at a stage gate on the basis of its own data. Until one does, the claim that the gates are real remains unevidenced. The events in Burkina Faso in 2025 show the other side of the same dynamic. A pathway can be interrupted not by its own criteria but by a change in the political conditions that made it possible. The programme's activities were suspended by national authorities amid a broader shift in the country's relationships with foreign-funded institutions. Nothing in the entomology changed. The lesson for design is that a phased pathway spanning fifteen years assumes fifteen years of stable regulatory and political conditions in a setting where that assumption has no historical support, and a programme that has not thought about what happens to released material, monitoring obligations, and data when it is asked to leave has not finished its planning. Writing gates that bite Three practices distinguish a programme with real stage gates from one with decorative ones. First, criteria are numerical and pre-registered before the stage's data are collected, in a document available to the regulator and, ideally, publicly. The numbers that matter most are the ones that would stop the programme, not the ones that would let it proceed: it is much easier to write conditions for success, and far more informative to write conditions for failure. Second, the body that assesses the gate is not the body that ran the stage. Independent data monitoring committees are standard in clinical trials and still unusual here; the few programmes that have them have benefited visibly. Third, the gate has an explicit "more data" option distinct from proceed and stop. The commonest real outcome of a well-run stage is that the estimate is too imprecise to support the next decision, and a pathway with only binary gates converts that into a proceed. Chapter 3: Problem Formulation and Environmental Risk Assessment Ask a room of biologists what could go wrong with releasing a gene-drive mosquito and you will get a list within a minute: it could jump to another species, it could collapse a food web, resistance could evolve, a worse vector could take over, it could spread to countries that never agreed to it. Ask what evidence would settle any of those, and the room goes quiet. That gap — between hazards that are easy to name and questions that are answerable — is what problem formulation exists to close, and it is the most under-appreciated methodological step in the entire pathway. Environmental risk assessment done badly consists of assembling a list of conceivable harms and writing a paragraph of reassurance under each. Done well, it is a structured exercise in converting vague concerns into testable propositions, ranked by how much they matter and how much they are in doubt, so that the testing programme is designed to resolve the ones that would change a decision. Everything downstream — which experiments are run, which endpoints are monitored, how long surveillance continues — is downstream of this step, and if it is done carelessly the rest of the programme measures the wrong things with great precision. Protection goals come before hazards The logic runs backwards from what is to be protected. A risk assessment cannot say whether an effect is harmful until someone has said what counts as harm, and that is not a scientific question. It is a policy question with a scientific implementation. Regulatory frameworks express this as protection goals: broad statements of what environmental policy aims to conserve — biodiversity, ecosystem services, non-target organisms, water quality, human health. Broad goals are not operational. The European Food Safety Authority's approach, developed for genetically modified plants and extended to insects, insists on converting them into specific protection goals with four components: the entity to be protected, the attribute of it that matters, the magnitude of change that would be unacceptable, and the temporal and spatial scale over which that magnitude is assessed. "Protect biodiversity" is not assessable. "No reduction greater than 20% in the abundance of insectivorous bat species within the release district, sustained over more than two breeding seasons" is assessable — one can argue about whether the number is right, which is exactly the argument that should be had, in public, before any data are collected. The conversion has an awkward political property that this field has mostly avoided confronting. Somebody has to choose the numbers, and the choice is a value judgement dressed in ecological clothing. In a national regulatory system with mature environmental legislation, the numbers come, ultimately, from statute and from the regulator's mandate. In settings where the relevant environmental law predates the technology by decades and contemplates nothing like it, they do not come from anywhere, and the applicant ends up proposing them — which is to say the party seeking approval defines what would count as unacceptable. That is a structural conflict of interest, and the honest way to manage it is to make the proposed specific protection goals the first thing published and consulted on, before the risk assessment that uses them. The pathway-to-harm discipline Given protection goals, the central analytic tool is the pathway to harm: an explicit causal chain from the release event to the damaged protection goal, with each link stated as a proposition that could be true or false. A pathway for horizontal transfer of a construct to a non-target species might run: the construct is present in released mosquitoes → released mosquitoes mate with, or are consumed by, individuals of species X → genetic material is transferred into the germline of species X → the construct is expressed and functional in species X → the construct spreads in species X → the spread reduces species X's abundance or alters its ecological function → a protection goal is damaged. Written out this way, the pathway does two things at once. It exposes which links are already known to be improbable, and it identifies where evidence would actually change the assessment. For interspecies transfer by predation, the third link is so improbable on everything known about vertebrate digestion and germline biology that no plausible evidence about the other links would alter the conclusion; the pathway is closed, and no further testing is warranted. For introgression between members of the Anopheles gambiae complex through hybridisation, the same link is not improbable at all — hybridisation happens at low but real rates, the sibling species are vectors themselves, and the pathway must be kept open and assessed with data. This is the discipline that distinguishes risk assessment from anxiety management. The question is never "could this happen?", because almost anything could. It is "is there a chain of events, each of which is plausible given what we know, that ends in a specified unacceptable outcome, and which link is most uncertain?" The systematic problem formulation conducted for a population-suppression drive in Anopheles gambiae, published in Malaria Journal in 2021, remains the best worked example in the literature. It enumerated pathways covering human and animal health, non-target organisms, biodiversity, and ecosystem function, and it found — as such exercises usually do — that most of the pathways that dominate public discussion close quickly on existing evidence, while the ones that stay open are narrower and less dramatic than the debate suggests: the ecological consequences of removing a substantial component of the insect biomass in specific habitats, the possibility of niche replacement by another vector, and the dynamics of resistance. The hazards that survive scrutiny It is worth being specific about which concerns hold up, because a risk assessment that treats all of them as equally live wastes the testing budget. Resistance to the drive. This is not a speculative hazard; it is an observed one. Nuclease-based drives generate resistance alleles when the cut site is repaired by non-homologous end joining rather than homology-directed repair. Alleles that destroy the target sequence while preserving gene function are the dangerous class: they are immune to the drive and are positively selected once the drive imposes a fitness cost, so they spread and the drive fails. Work in the Crisanti laboratory demonstrated this clearly in caged Anopheles populations in the 2010s, and it is the reason the doublesex target was chosen — a sequence so functionally constrained that most mutations destroying the drive's target site also destroy the gene. The methodological implication is that resistance monitoring must be sequence-level, must have a pre-release baseline of standing variation at the target site, and must be powered to detect alleles at low frequency, because a resistance allele that is at 0.5% when you first detect it is already a different problem from one at 0.005%. Niche replacement. If suppression of An. gambiae leaves the larval habitat available, something will use it. Whether the replacement transmits malaria better or worse is an empirical question with no general answer, and the arrival of An. stephensi in African cities makes it urgent rather than theoretical. Monitoring for this requires a sampling protocol that identifies all culicids to species, not just the target, and that runs long enough for replacement to occur — which is years, not a trial season. Food-web effects of suppression. Anopheles gambiae is one of hundreds of dipteran species in its habitats and, in most settings, not a dominant component of insect biomass. Reviews of predators and other species that might depend on it have generally found no specialist predator that would lose a food source it could not substitute. That is a real finding and it substantially closes the strongest version of this pathway, but it is not the same as showing there is no effect, and it is weaker in habitats where anopheline larvae are a large share of what lives in temporary rain-filled pools. This is a pathway to be assessed with site-specific data rather than dismissed with a general claim. Effects on the pathogen. Suppressing or modifying a vector exerts selection on Plasmodium. Whether that selection favours parasites transmissible by secondary vectors, or alters virulence, is genuinely uncertain and is one of the few pathways where no one has a confident prior. It is also one of the hardest to monitor, requiring parasite genomics in the affected region over long periods. Loss of immunity in the human population. Sustained reduction in transmission shifts the burden of clinical malaria to older age groups, because acquired immunity depends on repeated exposure. This is a well-characterised phenomenon from the history of malaria control, not a novel risk of genetic methods, and it argues for a control programme that goes to elimination rather than one that reduces transmission part way and stops — a consideration that belongs in the trial's exit planning. Uncertainty that cannot be resolved before the decision Every serious assessment in this field reaches a residue of irreducible uncertainty, and how it is handled is where technical judgement gives way to governance. Three distinctions help. Statistical uncertainty is the uncertainty of an estimate you can narrow with more data; it is the easy kind, and it is what confidence intervals express. Structural uncertainty is uncertainty about the model itself — whether the food-web model includes the right species, whether the dispersal kernel has the right functional form — and more data of the same kind does not reduce it; different data, or different models, are needed. Ignorance is the category of effects nobody has thought of, and the only honest response to it is monitoring designed to detect the unexpected rather than to confirm the expected, plus humility about the residual. The habit worth cultivating is to state, for each open pathway, which kind of uncertainty dominates. Where it is statistical, more field work is the answer, and the assessment should say how much. Where it is structural, the answer is model comparison and sensitivity analysis — show that the conclusion holds across plausible alternative model structures, or admit that it does not. Where it is ignorance, no amount of pre-release work helps, and the decision becomes explicitly one about acceptable exposure to the unknown, which is a political decision that should be labelled as such and taken by people with a mandate to take it. Tiered testing and the surrogate problem Chemical pesticide regulation resolved the "you cannot test everything" problem with tiered testing: begin with worst-case exposure of a small set of standard surrogate species in the laboratory, and escalate to more realistic, more expensive, more ecologically complete studies only for the endpoints where tier 1 gives a signal. It is an efficient design, and regulators reach for it here because it is what they know. It transfers poorly, for three reasons. First, the tier-1 logic depends on a dose concept. For a chemical, worst-case exposure means the highest concentration an organism could encounter, and if there is no effect at that concentration there is no need to model realistic exposure. A self-propagating organism has no concentration; the exposure is presence or absence of an interacting population, and "worst case" is not a dial you can turn up. Second, the standard surrogate species — a daphnid, a honeybee, a bobwhite quail — are chosen to represent taxonomic breadth for a substance that spreads through air, soil and water. The exposure route for a released mosquito is mating, predation, or competition, and the species that matter are the ones that actually mate with, eat, or compete with the target in the specific release habitat. The relevant list is site-specific and cannot be standardised. Third, in a tiered scheme, tier 1's conservatism is what licenses the shortcut. Here the conservative laboratory assay does not exist for most of the pathways that survive problem formulation, because those pathways are about population and community dynamics over years, and there is no bench assay for a food web. What replaces it, in practice, is a combination of case-specific tier 1 — laboratory and cage work on the specific interacting species identified by problem formulation at the site — and modelling as the escalation tier, with field monitoring designed to test the model's predictions rather than to observe everything. That is a weaker evidentiary structure than chemical regulation enjoys, and pretending otherwise by borrowing the vocabulary of tiers does not strengthen it. Risk assessment without benefit assessment is incoherent Almost every regulatory framework in this area assesses risk alone. The applicant must show that harm is unlikely; nowhere is the applicant asked to show that benefit is likely, and nowhere is the regulator asked to weigh them. For a product used voluntarily by an individual — a pesticide a farmer chooses to buy — that asymmetry is defensible, because the benefit judgement is devolved to the buyer. For a landscape intervention that nobody can opt out of, it is not. A decision that considers only the harms of acting, and not the harms of not acting, is systematically biased towards the status quo, and in a setting where the status quo is several hundred thousand malaria deaths a year that bias is not neutral. Making this coherent requires comparative assessment: the expected effects of the release against the expected effects of the realistic alternative, on the same protection goals, over the same period, with the uncertainty on both sides stated. This is difficult and contestable, and it is done nowhere in the current regulatory architecture except informally, in the political decision that sits on top of the technical assessment. The methodological recommendation is modest but real: a programme should produce the comparative analysis even where the regulator does not require it, because otherwise the only quantified document in the file is a catalogue of ways the intervention could cause harm, and that document will be read as the whole story. Who assesses, and who can review A risk assessment is only as good as the review it receives, and the reviewing capacity is the binding constraint in most of the countries where these trials would happen. The 2016 report of the US National Academies of Sciences, Engineering, and Medicine on gene drives made this point with some force: the phased pathway and the risk assessment framework both presuppose institutions capable of independent technical evaluation, and building that capacity is part of the work, not a precondition someone else supplies. Practically, this means three things. National biosafety authorities need reviewers who understand population genetics and spatial ecology, not only molecular biology, because the hard questions in a drive dossier are dynamical rather than constructional. They need access to independent expertise not funded by the applicant, which in a field with a small number of major funders is genuinely difficult and requires deliberate arrangement. And they need the standing to ask for data the applicant has not offered, including raw data and model code, rather than reviewing a summary. The last point deserves emphasis. Models are central to every risk argument in this field, and a model that cannot be re-run by the reviewer is a claim, not evidence. Requiring that simulation code and parameter files be submitted with the dossier, and that they run, is the single cheapest improvement available to regulators in this area. Several of the major modelling frameworks in use are open source, which makes the requirement reasonable rather than onerous. Practical failure modes Four errors recur in submitted risk assessments, and a reviewer can find them quickly. Comparator substitution. Assessing the release against a baseline of "no intervention" when the real-world alternative is continued insecticide use with its own substantial non-target effects — or assessing it against an idealised alternative that is not actually available. The comparator should be the realistic counterfactual, stated explicitly, and the assessment should be symmetric: the harms of the status quo belong in the frame. Scale mismatch. Assessing ecological effects at the scale of the release area when the organism operates at the scale of the region, or assessing them over the trial period when the effect has a decade-long lag. A drive's risk assessment cannot be bounded by the trial's boundaries, because the drive is not. Confusing absence of evidence with evidence of absence. "No adverse effects were observed" is a statement about the power of the monitoring, and it means nothing without the detection limit attached. An assessment should report, for each monitored endpoint, the smallest effect the monitoring could have detected. Treating the risk assessment as a one-time document. The assessment is a live model of the system that should be updated when field data arrive, including when they arrive from someone else's trial. In practice, submitted assessments are frozen at the moment of approval and never revisited, and the regulator has no mechanism to require revision. Building an explicit reassessment trigger into the approval — reassess on defined findings, or at defined intervals — is a small institutional change that would substantially improve the field. Hashtags: #CRISPRFieldTrials #GeneDriveResearch #GeneticVectorControl #Biosafety #EnvironmentalRiskAssessment #EcologicalModeling #PopulationGenetics #HomingGeneDrives #SelfLimitingMosquitoes #WolbachiaReplacement #VectorPopulationSuppression #PopulationModification #PhasedTestingPathway #FieldReleaseDesign #MarkReleaseRecapture #MosquitoDispersal #EntomologicalMonitoring #MolecularMonitoring #EcologicalRiskModeling #PathwayToHarm #ResistanceEvolution #NicheReplacement #TransboundaryGovernance #CommunityEngagement #FutureOfGeneDriveTrials

Download the Book (PDF): Introduction There are roughly seven thousand recognised rare diseases. Between four and five hundred of them have an approved therapy. The gap is not primarily a gap in biology. For a growing number of these conditions the causal gene is known, the mechanism is understood well enough to design a molecule, and the molecule can be made. The gap is in evidence: in the machinery by which a regulator comes to believe that a treatment works, and in the awkward fact that this machinery was designed for diseases with tens of thousands of patients and is being asked to operate on diseases with forty. The instinctive response is to argue about standards. One camp says that when a disease kills children before they reach school age, demanding a randomised placebo-controlled trial is a bureaucratic cruelty, and the bar should come down. The other camp points out, correctly, that most candidate drugs do not work, that uncontrolled observation is systematically biased toward optimism, and that approving ineffective therapies harms the very patients it is meant to help — by exposing them to toxicity, by consuming the small population that a genuinely effective competitor would need to test in, and by draining money and hope. Both camps have a point, and the argument between them has been running, largely unchanged, for forty years. It is the wrong argument. The standard of evidence is not the variable that should move. What should move — what has been moving, quietly and technically, since about 2010 — is the architecture of evidence: how information is gathered, from where, and how it is combined. That is the argument of this book. In a disease with forty patients, the conventional trial fails not because the standard is too high but because the design wastes information. It throws away everything known about how the disease progresses in untreated patients. It throws away what was learned in the adult population when the trial is in children, or in the common variant when the trial is in the rare one. It throws away the information contained in each patient's own pre-treatment trajectory. It answers one question with one protocol and then dismantles the infrastructure. It treats the boundary of the trial as the boundary of knowledge. When patients are abundant, that profligacy is affordable and buys something valuable — simplicity, transparency, and a design whose assumptions are so few that nobody has to argue about them. When patients are scarce, the same profligacy is the difference between an answerable question and an unanswerable one. The alternative is a family of designs that borrow information: from historical and natural-history data, from other arms of the same platform, from related subgroups, from other periods within the same patient, from routinely collected care data. Borrowing is not a trick and it is not a softening. It is a modelling commitment, and it comes with an obligation. Every borrowed unit of information rests on an assumption — that the historical control population is exchangeable with the concurrent one, that the response in one genetic subtype is informative about another, that a patient's disease is stable enough for a crossover to mean anything. Those assumptions can be false, and when they are false the borrowing imports bias rather than precision. The discipline that makes small-sample trial design respectable is the discipline of stating each assumption explicitly, quantifying how much information it is being asked to carry, pre-specifying what happens when the data contradict it, and demonstrating — by simulation, before a single patient is enrolled — how the design behaves across the range of worlds in which the assumption is wrong. That discipline is what distinguishes a Bayesian adaptive platform from an uncontrolled case series with better software. Both use fewer patients than a classical trial. Only one of them tells you, in advance and in numbers, what it will conclude when the drug does nothing. This is a practical book, written for people who have to make these decisions: statisticians designing rare disease programmes, clinicians and patient organisations who find themselves negotiating trial designs with sponsors, regulatory scientists assessing the resulting dossiers, and investors and programme managers trying to judge whether a proposed development plan is credible or merely fashionable. It assumes you know what a p-value is and what randomisation is for. It does not assume you have ever specified a prior distribution, run a trial simulation, or read a guidance document on externally controlled trials. The route runs as follows. The first two chapters establish the problem precisely: what actually breaks when sample size falls, in quantitative terms, and what regulators are actually asking for — which turns out to be considerably more flexible, and considerably more demanding about pre-specification, than the folklore suggests. The third chapter makes a claim that surprises people who come to this field expecting statistics: in most rare disease programmes, the endpoint, not the patient count, is the binding constraint, and effort spent on measurement buys more statistical power than effort spent on clever analysis. The middle of the book is about borrowing. Chapter four treats natural history data — registries, progression models, external control arms — as infrastructure rather than as a fallback, and sets out the conditions under which an external control is credible and the conditions under which it is a well-dressed guess. Chapters five and six build the Bayesian machinery: what a prior actually is in a regulatory context (a design parameter, subject to calibration, not a statement of personal belief), how to quantify how much information a prior contributes, and the specific constructions — power priors, commensurate priors, meta-analytic-predictive priors and their robust variants, hierarchical basket models — that let a design borrow strongly when the borrowed data agree with the new data and back away automatically when they do not. The last three chapters are about design architectures. Chapter seven takes the small-sample logic to its limit: the single patient as a complete randomised experiment, an idea with a long clinical pedigree that has become suddenly urgent now that therapies can be built for one person's mutation. Chapter eight scales in the opposite direction, to multi-arm multi-stage trials and master protocols — the insight that a shared control group and a standing infrastructure can make each new question far cheaper than the last, and the less-advertised costs in governance, error control, and operational complexity that come with it. Chapter nine is about what turns any of this into a regulatory submission: designing by simulation, controlling error rates you cannot compute in closed form, and assembling a dossier whose assumptions an assessor can audit. Three cautions before we start. The first is that none of these methods creates information. A Bayesian hierarchical model applied to twelve patients is still a model applied to twelve patients. What these designs do is extract more of the information that exists and combine it more efficiently, and there is a floor below which no amount of modelling helps. Recognising that floor is part of the craft. A design that promises to detect a modest effect in nine patients with adequate error control is almost always either assuming an enormous effect size, borrowing very heavily from somewhere, or wrong. The second is that the methods are not a menu from which one picks by taste. The design follows from the disease. A condition with a steep, well-characterised, near-deterministic decline — untreated infantile-onset spinal muscular atrophy, classic late-infantile neuronal ceroid lipofuscinosis — supports an external control in a way that a condition with a relapsing-remitting course and wide inter-patient variability never will. A chronic, stable, symptomatically treatable disease supports an N-of-1 crossover; a progressive irreversible one does not. Half the failures in this field come from applying a method that worked elsewhere to a disease whose natural history does not satisfy the method's preconditions. The third is that this is a field in motion. Regulatory positions that were informal in 2015 are written guidance now; positions that are informal today will be written guidance in five years. Where I cite a guidance document or a programme I have tried to name it precisely enough that you can find its current version, because the current version is the one that matters. The underlying statistical ideas move much more slowly than the regulatory vocabulary wrapped around them, and it is the ideas that this book is mainly about. One more thing. It is easy, in a technical treatment, to lose sight of what makes rare disease research different from a difficult inference problem. The population is small enough that a trial design decision is not an abstraction: choosing a two-arm placebo-controlled design in a disease with sixty diagnosed patients means that thirty identifiable children receive placebo, and it may mean that no second trial of anything is possible for five years because the eligible population has been consumed. Efficiency here is not a matter of budget. Every design decision allocates a scarce and irreplaceable resource, and the people who constitute that resource generally know it better than the sponsor does. That is a good reason to get the statistics right, and a good reason to involve patient organisations in design decisions long before the protocol is drafted. It is not a reason to pretend that a weak design is a strong one. Chapter 1: The Arithmetic of Scarcity Start with the number that governs everything else. For a two-arm parallel trial comparing means, with a two-sided significance level of 0.05 and 80% power, the required sample size per arm is approximately 15.7 divided by the square of the standardised effect size — the difference in means divided by the within-group standard deviation. The relationship is inverse-square, and the inverse-square is the whole problem. Halving the effect you want to detect quadruples the patients you need. Doubling the outcome's variability quadruples them again. Table 1 sets out what this means in practice. Table 1. Sample size per arm and attainable power for a two-arm parallel trial comparing means, two-sided α = 0.05, by standardised effect size. Computed from the normal approximation n = 2(z₀.₉₇₅ + z₁₋β)²/δ². Standardised effect (δ) n per arm, 80% power n per arm, 90% power Power with 15 per arm 0.3 175 234 13% 0.5 63 85 28% 0.8 25 33 59% 1.0 16 21 78% 1.5 7 10 98% 2.0 4 6 >99% Read the right-hand column carefully, because it is the condition most rare disease programmes actually find themselves in. Thirty patients, randomised one-to-one, gives a well-conducted trial a 28% chance of detecting a treatment effect of half a standard deviation. Half a standard deviation is not a trivial effect; in many chronic diseases it would be considered clinically important and would change practice. The trial would miss it nearly three times in four. Worse, when such a trial does reach significance, the observed effect is necessarily large — you cannot cross the threshold with a small estimate when the standard error is wide — so the published estimate is biased upward, sometimes by a factor of two. This is the winner's curse, and it is why small positive trials so often fail to replicate. A field that runs many underpowered trials does not merely learn slowly; it accumulates a literature of overstated effects. The conventional response is to hunt for a larger δ. This is not cheating — a therapy that replaces a missing enzyme or corrects a single causal gene may genuinely produce an effect of one or two standard deviations, and the bottom rows of the table are where gene and enzyme replacement therapies often live. Onasemnogene abeparvovec in infantile spinal muscular atrophy, where untreated patients essentially never achieve independent sitting and treated infants frequently do, is operating at an effect size where a dozen patients is genuinely sufficient. But assuming a large δ because you need one is the commonest self-deception in this field. The assumed effect size in a rare disease protocol is often reverse-engineered from the number of patients available, then written into the statistical section as though it had come from the biology. When the trial fails, everyone blames recruitment. Where the variance comes from If δ is a ratio, the denominator deserves as much attention as the numerator, and it usually gets far less. Three sources of variance dominate in rare diseases, and each is attackable. The first is genuine biological heterogeneity. Rare diseases are frequently defined by a gene rather than a phenotype, and a single gene can produce a spectrum. In Duchenne muscular dystrophy, the position of the deletion determines which exon-skipping therapy is applicable and influences residual dystrophin. In Pompe disease, infantile-onset and late-onset forms differ so profoundly in trajectory that pooling them would be indefensible. In cystic fibrosis, the class of CFTR mutation determines whether a modulator can work at all. When a trial enrols across such a spectrum, the within-group standard deviation absorbs the between-phenotype differences, and δ collapses. Narrowing eligibility raises δ but shrinks the eligible population — the central trade-off of rare disease enrolment, and one that should be made deliberately with a simulation rather than by instinct. The second is age and stage. Most rare diseases are progressive, and the rate of progression typically varies with baseline severity in a non-linear way. A patient with a forced vital capacity of 90% has room to decline; one at 35% is near the floor of the instrument and may look stable because the measure has nowhere left to go. Enrolling both in the same trial, with change from baseline as the endpoint, produces a variance dominated by where patients started. Stratified randomisation helps, and so does covariate adjustment, but the more fundamental fix is to define eligibility by a window of disease stage rather than by diagnosis alone. The third is measurement error, and it is the one most often ignored. Many rare disease endpoints are observer-rated functional scales administered by clinicians who may see two patients with the condition in a career. A six-minute walk test depends on encouragement, corridor length, time of day, and whether the child slept. A motor function scale with seventeen items rated 0-2 by an untrained assessor can have a test-retest variability comparable to a year of disease progression. Because variance enters the sample size formula as a square, halving measurement noise has the same effect on required sample size as doubling the treatment effect. Central raters, video adjudication, duplicate baseline assessments averaged together, and rigorous assessor certification are not administrative niceties. They are among the cheapest sources of statistical power available, and they are usually the first thing cut from a stretched budget. The three constraints that do not appear in the formula Sample size arithmetic assumes that the patients exist, can be found, and will consent. In rare disease, each of those assumptions carries its own failure mode. Prevalence is not availability. A disease with a stated prevalence of one in a hundred thousand implies about eighty patients in the United Kingdom and around three thousand in the European Union. The number actually reachable is far smaller. Some are undiagnosed; diagnostic odysseys in rare disease still commonly run five to seven years, and for many conditions the majority of prevalent cases have never been correctly labelled. Some are outside the eligibility window by age or stage. Some are in countries where the trial will not run. Some cannot travel to a site every fortnight for infusions, which for a disease causing progressive immobility is a sizeable fraction. Some are already enrolled in a competitor's trial, and for an ultra-rare condition with two active programmes, the second sponsor may find that the population has been consumed. A realistic feasibility estimate typically lands at ten to twenty per cent of prevalent cases, and sponsors who plan from prevalence rather than from confirmed, contactable, eligible patients routinely find themselves two years into an eighteen-month enrolment. Time is an active ingredient. In a common disease, extending enrolment from one year to three is an expense. In a rare disease it changes the trial. Standard of care may shift midway, as it did repeatedly in spinal muscular atrophy and Duchenne once the first therapies arrived, so that patients enrolled in year three have a different background regimen from those enrolled in year one. Diagnostic criteria and genetic panels improve, so later enrollees are milder. Newborn screening programmes, once implemented, transform the incident population from symptomatic infants to presymptomatic ones — a change so large that it invalidates the natural history data collected a decade earlier. A long enrolment period silently introduces drift, and drift is precisely the thing that breaks the exchangeability assumptions on which historical borrowing depends. It is worth saying plainly: slow enrolment does not merely delay an answer, it degrades the question. Geography multiplies operational variance. Assembling forty patients usually means twenty sites in eight countries, which means eight regulatory submissions, eight translations of the outcome measure, eight sets of local practice, and a per-site enrolment of two. With two patients per site, no site develops proficiency in the assessment, no site's principal investigator gains a feel for the protocol's ambiguities, and a single site's protocol deviation can move the overall result. The relationship between site count and data quality in rare disease is close to inverse, and this is a strong argument for centralising assessments at a handful of expert centres even when it means flying patients in. Why the standard fixes do not work Faced with an underpowered design, the standard toolkit offers three moves. Each is less useful here than it looks. Relax the significance level. Testing at one-sided 0.10 rather than two-sided 0.05 does buy power. It also buys false positives, and in a therapeutic area where a single approval may define standard of care for a generation and effectively foreclose further trials, a doubled false positive rate is expensive in a way that is hard to reverse. Regulators have in practice accepted relaxed alpha in some rare disease settings, but they have done so as part of a negotiated package — usually alongside stronger evidence elsewhere in the dossier — not as a free parameter. The deeper problem is that relaxing alpha does nothing about the winner's curse. A trial powered at 40% still produces inflated effect estimates when it wins, whatever threshold it crossed. Use a surrogate endpoint with lower variance. This genuinely works, and Chapter 3 is largely about it. But a surrogate buys power only to the extent that it is a valid stand-in for clinical benefit, and validating a surrogate normally requires exactly the large trials that rare diseases cannot run. The result is a circularity that the accelerated approval pathway exists to manage — and manages imperfectly, as the long argument over dystrophin expression in Duchenne demonstrates. Enrich the population. Restrict to patients most likely to respond, and δ rises. This is sound, and it is the logic behind biomarker-defined eligibility. The cost is that the trial then answers a narrower question, and the label follows the trial. Enrichment also shrinks the accessible population, so it trades power against feasibility on the same axis you were trying to escape. None of these is useless; all of them are incremental. They tune a design whose fundamental structure — one question, one cohort, information from nowhere else — is the thing that does not fit. The cost of an uninterpretable result There is a further consequence of underpowering that the sample size formula does not express, and in rare disease it may be the most important one. In a common disease, a negative trial is information. The candidate failed, the field moves on, and another trial can be run next year in a different population with a different molecule. In a rare disease, an underpowered negative trial is frequently not information at all. A trial with 30% power that returns a non-significant result has told you almost nothing: the probability of that result was high whether the drug worked or not. But the trial has consumed the eligible population for two to four years, exhausted the sponsor's appetite, and — because negative results attach to mechanisms as well as to molecules — often discouraged investment in the entire target. Several rare disease mechanisms have been abandoned on the basis of single small trials whose confidence intervals comfortably included clinically important benefit. The asymmetry runs the other way too. An underpowered trial that reaches significance produces an effect estimate inflated by the winner's curse, and in rare disease that estimate is unlikely ever to be corrected, because the confirmatory trial that would correct it cannot recruit once a therapy is available. The inflated estimate then propagates into the label, into the health technology assessment, into the cost-effectiveness model, and into the expectations of families. When the real-world effect turns out to be half the trial estimate, the resulting disappointment is read as a failure of the therapy rather than as a predictable property of the design that produced the number. Both failures are avoidable, and neither is avoided by running the trial and hoping. They are avoided at the design stage, by being honest about what a given structure can and cannot detect, and by changing the structure when the honest answer is "nothing useful." What the structure wastes Consider a hypothetical but entirely typical situation. A progressive paediatric neurodegenerative disease has about two hundred diagnosed patients worldwide. A registry has followed ninety of them for up to eight years, with annual assessments on the same functional scale the trial will use. The decline is monotonic and, within a defined age band, reasonably predictable: patients lose about six points a year with a between-patient standard deviation of two. A drug is ready. Forty patients can plausibly be enrolled over two years. The conventional design randomises twenty to drug and twenty to placebo and compares twelve-month change. Consider what it discards. It discards the ninety registry patients, whose trajectories tell you a great deal about what the placebo arm will do — arguably more than twenty concurrent placebo patients will, since the registry patients have been followed longer and are more numerous. It discards each trial patient's own pre-randomisation trajectory, which for a progressive disease with a diagnostic delay is often two or three years of retrospective data establishing that patient's individual rate of decline. It discards the information in the dose-finding cohort that preceded the trial. It discards, by design, the possibility of learning anything about the second candidate molecule that the same sponsor has in preclinical development, because the trial is a closed system that will be dismantled on readout. And it allocates twenty children to placebo in a fatal disease, which the families will accept only if they are persuaded there was no alternative. Now consider the information that structure could have used. The registry data can supply a prior distribution for the control-arm decline rate, allowing the concurrent control arm to be reduced — not eliminated, but reduced, perhaps to a two-to-one or three-to-one allocation, so that two-thirds or three-quarters of participants receive active drug. Each patient's own pre-treatment slope can be used as an internal control in a within-patient comparison, which removes between-patient variance entirely from that component of the analysis. The trial infrastructure, if built as a platform, can accept the second molecule when it is ready, sharing the control group across both comparisons. And the whole design can be specified so that if the registry data turn out to disagree with the concurrent controls — if the trial's placebo patients decline at four points a year rather than six — the borrowing automatically diminishes and the analysis falls back toward the concurrent comparison. Each of those moves is a subject of a later chapter. The point here is only that they are structural, not analytical. You do not get them by choosing a better test statistic at the end. You get them by designing a different trial, and the design decision has to be made before anyone is enrolled. The floor It would be dishonest to leave the impression that structure solves everything. There is a floor, and it is worth being explicit about where it sits. Information borrowed from outside the trial is only as good as the exchangeability of its source. If the registry patients were assessed with a different version of the scale, or diagnosed later in their course, or managed before the introduction of a supportive therapy that has since become standard, then borrowing from them imports a bias that no amount of statistical machinery removes — it merely relabels it. Methods that down-weight borrowing when a conflict is detected help, but detection itself requires data: with eight concurrent controls you have limited ability to notice that your historical prior is wrong, and the methods that are meant to protect you are at their weakest exactly when you most need them. Within-patient designs are similarly conditional. A crossover requires that the treatment effect appear and disappear on a timescale short relative to disease progression, and that carryover can be excluded. For a disease-modifying therapy in a degenerative condition, both conditions fail, and no design converts a progressive irreversible disease into one that supports crossover. And no design overcomes an effect that is not there. Much of the apparatus in this book is aimed at detecting true effects of moderate size with fewer patients. None of it improves the prior probability that the drug works, which in rare diseases — despite strong genetic rationale — remains well below one half. So the honest statement of the problem is this. In a disease with a few dozen patients, the classical two-arm trial is not merely inconvenient; for any effect smaller than about one standard deviation it simply cannot deliver a reliable answer, and pretending otherwise produces a literature of overstated positives and uninterpretable negatives. The response is not to lower the standard of proof. It is to build designs that use every source of information the disease affords, to state the assumptions under which each source is being used, and to demonstrate in advance how the design behaves when those assumptions fail. The rest of this book is about how to do that, and the next chapter is about the audience that will have to be convinced. Chapter 2: What the Regulator Is Actually Asking A persistent piece of folklore holds that regulators demand two adequate and well-controlled trials, that anything less will be rejected, and that rare disease development is therefore a matter of persuading an inflexible agency to make an exception. Every part of this is wrong, and believing it leads sponsors to design the wrong trial and then negotiate from a weak position. The US statutory standard is "substantial evidence of effectiveness," established by the 1962 Kefauver-Harris amendments. The phrase has always been interpreted through the concept of adequate and well-controlled investigations, and the governing regulation — 21 CFR 314.126 — lists five acceptable types of control: placebo concurrent control, dose-comparison concurrent control, no treatment concurrent control, active treatment concurrent control, and historical control. Historical control has been in the regulation since 1985. The regulation adds the observation that historically controlled trials are "reserved for special circumstances," and gives as an example diseases with high and predictable mortality. That is a description of a substantial share of the ultra-rare disease landscape. The requirement for more than one trial is likewise softer than the folklore. FDA guidance has for decades allowed that a single adequate and well-controlled trial plus confirmatory evidence may suffice, and in 2019 the agency issued draft guidance specifically on what counts as confirmatory evidence in that construction, finalised in subsequent revision. Confirmatory evidence can include mechanistic data, evidence from a related indication, natural history data, or compelling results from an independent cohort within the same programme. In practice, the overwhelming majority of rare disease approvals in the last decade rest on one trial. The European position is set out most directly in the CHMP Guideline on clinical trials in small populations (CHMP/EWP/83561/2005), adopted in 2006 and still the anchoring document. Its central statement is that there is no special evidentiary standard for small populations: the same level of evidence is required, but the methods used to generate it may differ, and the regulator will weigh the totality. More recently the EMA has issued a reflection paper on establishing efficacy on the basis of single-arm trials submitted as pivotal evidence, which is notable less for permitting single-arm trials — they have been accepted for years — than for setting out systematically the conditions under which one is interpretable and the analyses an applicant is expected to provide. So the question is not whether flexibility exists. It does, and it is written down. The question is what the flexibility is purchased with. The currency is pre-specification Every piece of design flexibility a regulator grants is paid for in advance commitment. This is the single most important thing to understand about rare disease regulatory strategy, and it inverts the intuition that small trials permit more post hoc reasoning because there is less data to go around. The logic is straightforward. In a large randomised trial, randomisation and size together limit how much a sponsor's choices can influence the result. If you run a two-thousand-patient trial with a pre-specified primary endpoint, the assessor can be fairly confident that the effect estimate reflects the drug rather than the analysis. In a thirty-patient externally controlled trial, almost every number in the submission is a choice: which historical patients were eligible, which covariates were adjusted for, which time point was primary, how missing data were handled, what counted as a responder. Each of those choices, made after seeing the data, can move the result more than the drug does. The assessor's only protection is that they were made before. This is why the practical difference between a credible small trial and an uninterpretable one is often nothing to do with the statistics and everything to do with the timestamp. An externally controlled analysis with a propensity model specified, locked, and submitted before the trial data were unblinded is evidence. The identical analysis specified afterward is a hypothesis. Assessors are explicit about this distinction, and sponsors who do not build their programmes around it find that their most sophisticated analyses are discounted. The corollary is that the regulatory interaction has to happen early. The instruments that exist for this — pre-IND and Type B meetings, EMA Scientific Advice and Protocol Assistance, the parallel EMA-HTA advice procedure, and for novel designs the FDA's Complex Innovative Trial Design Paired Meeting Program — are not formalities. For a Bayesian design with borrowing, the agency will want to see the prior, the borrowing mechanism, and the simulation report, and will frequently ask for changes. Discovering those changes at submission is fatal; discovering them at protocol stage is a Tuesday. The instruments available Table 2 summarises the main regulatory instruments a rare disease programme can use, what each permits, and what each costs. Table 2. Principal regulatory instruments for evidentiary flexibility in rare disease development. Instrument Jurisdiction What it permits What it requires in return Orphan designation US, EU, JP, others Fee relief, protocol assistance, market exclusivity (7 yr US, 10 yr EU) Prevalence threshold; no evidentiary change Accelerated approval US Approval on a surrogate reasonably likely to predict benefit Confirmatory trial, generally underway at approval; withdrawal procedures Conditional marketing authorisation EU Approval on less complete data where unmet need is high Specific obligations with deadlines; annual renewal Single-arm pivotal trial US, EU No concurrent control arm Predictable natural history, large effect, objective endpoint, pre-specified external comparison Externally controlled trial US, EU Formal comparison to historical or registry cohort Fit-for-purpose data, documented comparability, pre-specified adjustment, sensitivity analyses Complex innovative design support US (CID programme) Bayesian and adaptive designs, borrowing Full simulation report; public disclosure of the design case Two features of this table deserve comment. First, orphan designation is not an evidentiary instrument. It is an economic one, and the commonest strategic confusion in the field is to treat designation as though it implied a relaxed standard of proof. It does not. A designated product is assessed on the same statutory standard as any other. Second, accelerated approval and conditional marketing authorisation are the instruments that actually change what must be proven at the time of approval, and they are the ones under most pressure. The eteplirsen approval in 2016 — granted on the basis of a small increase in dystrophin expression, over the objection of the review division — became the reference case for arguments on both sides of the debate about surrogate-based approval in Duchenne muscular dystrophy. The subsequent difficulty in completing confirmatory trials for accelerated approvals, in oncology as well as rare disease, produced legislative tightening in the 2022 omnibus reform, which gave the FDA authority to require that confirmatory trials be underway before accelerated approval is granted and streamlined withdrawal procedures. The practical message for a sponsor designing a rare disease programme in 2026 is that the confirmatory trial is now part of the initial design problem, not a later concern, and that a development plan in which the confirmatory evidence is undefined will attract attention. Estimands: saying exactly what you mean The ICH E9(R1) addendum, adopted in 2019, introduced the estimand framework, and although it was not written with rare diseases in mind it matters disproportionately here. An estimand is a precise specification of the treatment effect being estimated: the population, the variable, the handling of intercurrent events, the population-level summary, and the treatment conditions being compared. The addendum's insistence is that this be settled before the analysis, and that the strategy for each intercurrent event be named explicitly. Rare disease trials are unusually full of intercurrent events. Patients start rescue medication. Patients receive a transplant. Patients lose ambulation and the walk test becomes inapplicable. Patients die, and death is both an intercurrent event and, often, the outcome of real interest. In a trial of thirty patients, three such events are ten per cent of the evidence, and the choice between a treatment policy strategy (analyse regardless of what happened afterward), a hypothetical strategy (estimate what would have happened had the event not occurred), and a composite strategy (treat the event as a bad outcome) frequently determines whether the trial is positive. Working through the estimand properly has a second benefit that is easy to miss: it usually clarifies the design. If the intercurrent event of interest is death, and the composite strategy is chosen, then a rank-based composite endpoint follows naturally and the design should be built around it. If the hypothetical strategy is chosen, then the analysis depends on untestable assumptions about counterfactual trajectories, and the sponsor needs natural history data good enough to support them. Deciding the estimand first, and the design second, is an ordering that prevents a great deal of trouble. What assessors actually worry about Having sat on both sides of these discussions, I would summarise the assessor's concerns about a small-sample design under four headings. Is the comparison fair? For an externally controlled or historically borrowed design, this is the dominant question. Were the external patients diagnosed by the same criteria? Assessed with the same instrument, by comparably trained raters, at comparable intervals? Managed with the same supportive care? Selected without knowledge of outcome? Would an external patient have been eligible for the trial, and is that eligibility assessed on data available at the corresponding index date rather than retrospectively? Most external control submissions that fail, fail here, and they fail on data provenance rather than on statistical method. How much is the prior doing? For Bayesian designs, assessors want a number, not a philosophy. The question "how many patients' worth of information is this prior contributing?" has a technical answer — prior effective sample size — and an applicant who cannot produce it will be asked for it. A prior contributing the equivalent of thirty historical patients to a trial with fifteen concurrent controls is making a strong claim, and the strength of that claim needs to be visible rather than buried in a hyperparameter. What happens if you are wrong? This is the simulation question. For any design whose error rates cannot be derived in closed form — which is to say every adaptive or borrowing design — the agency expects a simulation report covering the null case, a range of alternative effect sizes, and, critically, scenarios in which the design's assumptions are violated: the historical control rate has drifted, the subgroups are not exchangeable, the enrolment is slower than planned. The type I error under prior-data conflict is the number that decides many of these discussions. Chapter 9 treats this in detail. Can we tell what you would have done? Adaptive designs create decision points, and each decision point is an opportunity for discretion. A protocol that says the sponsor "may" drop an arm at interim on the basis of "insufficient activity" is not pre-specified; one that says an arm is dropped when the posterior probability that its effect exceeds zero falls below 0.10 at a pre-defined interim is. Assessors read these clauses closely, and vagueness at a decision point is treated as an opportunity for bias, which is exactly what it is. The second audience Regulatory approval is not the end of the evidentiary problem, and designing only for the regulator is a mistake that rare disease programmes make with some regularity. Health technology assessment bodies and payers apply a different standard, and the difference is not one of strictness but of question. The regulator asks whether the therapy works and is acceptably safe. The payer asks how much health benefit it delivers relative to its cost, over a lifetime horizon, compared with existing care. Answering that question requires quantities a registrational trial is not designed to produce: the absolute magnitude of benefit on outcomes that map to quality-adjusted survival, the durability of that benefit beyond the trial's duration, and the untreated counterfactual over decades. For rare disease therapies, often priced very high on the argument that development costs are spread over few patients, this second assessment is frequently the harder one. Several therapies approved on the strength of a surrogate or a small externally controlled trial have then spent years in reimbursement negotiation, because the evidence sufficient to establish that the drug works was not sufficient to establish how much it is worth. The design implications are concrete and cheap to act on if considered early. Collect a preference-based quality-of-life instrument alongside the clinical endpoints, even when it will not be a registrational endpoint, because retro-fitting utilities from clinical scales is contentious and weak. Plan long-term follow-up of treated patients from the outset, ideally through the same registry that supplied the natural history data, so that durability is observed rather than extrapolated. Model the untreated lifetime trajectory from the natural history cohort, which the programme already has. And use the parallel scientific advice procedures that let a sponsor consult regulator and assessment bodies together, which exist precisely because the two sets of requirements are easier to satisfy jointly than sequentially. Flexibility that is real, and flexibility that is not It is worth separating the areas where regulators have demonstrably moved from those where they have not. They have moved substantially on control strategy. External controls, single-arm designs with pre-specified natural history comparisons, and unequal randomisation ratios that favour active treatment are all routinely accepted in appropriate settings. They have moved on adaptive features: sample size re-estimation, seamless phase transitions, arm dropping and adding within master protocols are all covered by written guidance. They have moved on Bayesian borrowing, though more cautiously for drugs than for devices, where the Center for Devices and Radiological Health has accepted Bayesian designs with informative priors for two decades. They have moved considerably on extrapolation, particularly from adults to children: the presumption now runs toward extrapolating where the disease and exposure-response relationship are sufficiently similar, with paediatric studies confirming pharmacokinetics and safety rather than repeating efficacy. They have created dedicated infrastructure — the Rare Disease Endpoint Advancement pilot, which pairs sponsors with the agency to develop novel efficacy endpoints, and the Support for clinical Trials Advancing Rare disease Treatment pilot, offering frequent ad hoc communication to a small number of selected programmes. They have not moved on randomisation where randomisation is feasible. A sponsor who proposes a single-arm design in a disease where a concurrent control could be assembled will be asked why, and "recruitment is difficult" is not an answer that survives contact with an assessor who can count the patients in the registry. They have not moved on blinding where blinding is feasible; open-label assessment of a subjective functional endpoint in a trial patients desperately want to succeed is a known and quantified source of bias, and the availability of central video adjudication removes most of the practical objections. They have not moved on data integrity, and the sloppiness that a small single-centre programme can accumulate — undocumented protocol deviations, assessments performed outside window, source data that cannot be reconciled — is more damaging here than in a large trial because there is no volume of clean data to dilute it. And they have not, despite appearances, moved on the underlying standard. The 2023 final FDA guidance on rare disease drug development, which consolidates much of the agency's thinking, is a document about how to generate adequate evidence efficiently, not about accepting less of it. Read it as a design manual rather than as a concession and it is considerably more useful. The practical implication The strategic error that costs rare disease programmes the most is designing a conventional trial, discovering it will not recruit, and then seeking regulatory relief from a position of weakness — with a protocol already written, sites already open, and no simulation work to support an alternative. The alternative is to treat the evidentiary architecture as the first design decision rather than the last. What is the estimand? Given that estimand and the disease's natural history, what is the most informative control the disease permits? Where does the information to support that control come from, and is it fit for purpose? How much borrowing does the design require, and how does it behave when the borrowed information is wrong? Those questions are answerable eighteen months before first patient in, and answering them is what the early regulatory interaction is for. Everything in the remaining chapters is in service of answering them. The next question in sequence is one that sponsors habitually skip: before deciding how to compare, decide what to measure. Chapter 3: The Endpoint Is the Binding Constraint Ask a rare disease sponsor what limits their trial and they will say patients. Look at the programmes that fail, and a large share of them had enough patients and the wrong measurement. The reason returns to the arithmetic of the previous chapter. Power depends on the standardised effect size, which is the treatment effect divided by the outcome's variability. The endpoint determines both terms. An endpoint that responds strongly to the drug raises the numerator; an endpoint measured precisely lowers the denominator. Changing the endpoint can therefore change the required sample size by a factor of five or ten, and unlike patient availability, it is under the sponsor's control. Consider two versions of the same trial in a progressive neuromuscular disease. Version one uses change in a seventeen-item clinician-rated motor scale at twelve months, administered by local physiotherapists at eighteen sites, with a between-patient standard deviation of change of 4.5 points and an anticipated treatment effect of 2 points. That is δ = 0.44, requiring about eighty patients per arm for 80% power. Version two uses the same scale, but with assessors centrally certified, each patient's baseline taken as the average of two assessments two weeks apart, video review of all assessments by two independent blinded raters, and analysis by mixed model with baseline value and age as covariates. The measurement improvements plausibly reduce the standard deviation of change to 3.0; the covariate adjustment reduces residual variance further. Now δ is around 0.67 and about thirty-five patients per arm suffice. Nothing about the drug changed. The trial went from impossible to difficult. This is the highest-leverage work in rare disease trial design, and it is systematically under-resourced because it happens early, costs money in a phase where budgets are tight, and produces no visible deliverable. What makes an endpoint work in a small trial Four properties matter, roughly in this order. Sensitivity to change over the trial duration. The endpoint must move, in untreated patients, over the period you can afford to observe. In a slowly progressive disease, a twelve-month trial on an endpoint that changes by two points a year with a measurement error of three points is measuring noise. The fix is either a longer trial, a more sensitive measure, or an enrichment strategy that selects patients in the steepest phase of their trajectory. Natural history data are what tell you which phase that is, which is one of several reasons the natural history study should precede the trial by years rather than run alongside it. Freedom from floor and ceiling effects. Ordinal functional scales were mostly designed for clinical description, not for measuring change, and they compress at the extremes. A patient scoring 2 out of 2 on every item of a motor scale cannot improve on that instrument regardless of what the drug does; a patient at zero cannot decline. In a rare disease population spanning a wide severity range, a substantial fraction of the enrolled patients may be at a floor or ceiling for the primary measure, contributing nothing but variance. Checking the distribution of baseline scores in the natural history cohort against the instrument's range is a fifteen-minute exercise that has saved programmes. Objectivity under open-label conditions. Many rare disease trials cannot be fully blinded — an infusion with a distinctive reaction profile, a gene therapy with a visible administration procedure, a crossover the patient can feel. Endpoints that depend on effort or on rater judgement will move under those conditions whether or not the drug works. Timed function tests depend on encouragement; global impression scales depend on hope. Where blinding is imperfect, prefer endpoints that are instrumented, laboratory-measured, or adjudicated by raters with no contact with the patient. Regulatory acceptability as a measure of benefit. An endpoint can be precise, sensitive and objective and still not be accepted as evidence of clinical benefit. This is where sponsors most often discover their problem late. The remedy is the formal route: qualification of a drug development tool in the US, or the EMA qualification of novel methodologies procedure, and in the rare disease setting specifically the Rare Disease Endpoint Advancement pilot, which was created because the agency recognised that most rare diseases have no validated outcome measure and that sponsors were being asked to validate one without the population to do it in. Building an endpoint when none exists For most of the seven thousand rare diseases, no validated outcome measure exists. Borrowing one from an adjacent condition is the default, and it is often wrong. A scale developed for adult multiple sclerosis measures things that do not limit a four-year-old with a lysosomal storage disorder; a paediatric quality-of-life instrument validated in oncology asks about domains a child with a congenital myopathy has never experienced differently. The disciplined approach follows the clinical outcome assessment development pathway, compressed. It starts with qualitative work: structured interviews with patients and caregivers to establish which symptoms and functional limitations matter most and how they are experienced. This concept elicitation step is not a courtesy. In several conditions it has overturned the assumed primary endpoint — most famously in conditions where clinicians had focused on ambulation while families identified upper-limb function, fatigue, or continence as the dominant burden. It then moves to item generation, cognitive debriefing to confirm that patients understand the items as intended, and psychometric evaluation in the natural history cohort: test-retest reliability, known-groups validity, and, crucially, estimation of a meaningful within-patient change threshold. That last quantity deserves emphasis because it is what converts a continuous endpoint into a responder definition, and responder definitions are attractive in small trials for a reason that is mostly illusory. A binary responder endpoint is easier to explain and appears to sidestep questions about what a two-point change means. But dichotomising a continuous measure throws away information — typically the equivalent of losing a third of the sample — and in a small trial that loss is unaffordable. The better use of a meaningful-change threshold is as a supporting analysis: report the continuous primary, and support it with the proportion exceeding the threshold as a clinically interpretable secondary. Reversing this ordering is common and costly. Where the disease is so heterogeneous that no common endpoint applies to all patients, goal attainment scaling offers a genuine alternative. Each patient, with their clinician and family, defines individualised, pre-specified goals with graded levels of achievement, set before randomisation and before treatment assignment is known. The outcome is the patient's position on their own scale. This handles heterogeneity that no fixed instrument can, and it has been used in conditions where patients' limitations share no common axis. Its weaknesses are real — goal setting is subjective, blinding is hard, and comparability across patients rests on the calibration of the graded levels — so it works best with independent goal review, pre-specified goal templates by domain, and blinded scoring. It is not a soft option, and regulators have engaged with it seriously where it is well constructed. Composite and rank-based endpoints Rare diseases frequently produce a problem that single endpoints handle badly: the outcomes that matter most are rare, and the outcomes that are common are less important. Death is unambiguous but happens to four patients; functional decline happens to everyone but is harder to interpret. A conventional composite that counts "death or decline" as a binary event treats a death and a two-point decline as equivalent, which nobody believes. Rank-based composites solve this by imposing a hierarchy. Every pair of patients is compared: first on the most important outcome, and only if that does not discriminate, on the next. The Finkelstein-Schoenfeld approach, developed for combining mortality with longitudinal measures, compares each patient against every other — if one died and the other did not, the survivor wins; if both survived, the comparison falls to the functional measure. The win ratio, a closely related construction popularised in cardiology, forms the same pairwise comparisons and reports the ratio of wins to losses, which has the advantage of being interpretable as a quantity rather than only as a test. These methods are well suited to rare disease for three reasons. They use the clinically important but rare outcome without requiring enough events to power a trial on it alone. They preserve the ordering that clinicians and families actually hold. And they are relatively robust, being rank-based, to the skewed distributions and outliers that small samples produce. The costs are that the estimand is subtle — a win ratio is not a hazard ratio and does not have a simple counterfactual interpretation — and that the result depends on the hierarchy, which must therefore be pre-specified and defensible. Involving patient representatives in setting the hierarchy is both good practice and good strategy, since a hierarchy that reflects stated patient priorities is far easier to defend than one constructed by a statistician. Biomarkers, surrogates, and the accelerated approval problem The most powerful variance reduction available is usually a biomarker, and the most contested question in rare disease regulation is when a biomarker may substitute for clinical benefit. The distinction that matters is between a biomarker used as a pharmacodynamic measure and one used as a surrogate endpoint. A pharmacodynamic biomarker demonstrates that the drug is doing what it was designed to do — that the enzyme is present, the substrate is cleared, the transcript is skipped. It supports dose selection and provides mechanistic confirmation, and it is essentially uncontroversial. A surrogate endpoint is a biomarker used as a substitute for a clinical outcome, on the basis that change in the biomarker predicts change in how a patient feels, functions, or survives. That is a much stronger claim, and establishing it normally requires demonstrating that the treatment effect on the biomarker accounts for the treatment effect on the clinical outcome — which requires trials large enough to observe both. The accelerated approval pathway exists precisely because that requirement is circular in rare disease. It permits approval on a surrogate "reasonably likely to predict" benefit, with a confirmatory trial to follow. Two cases illustrate the range of outcomes. Eteplirsen, approved for Duchenne muscular dystrophy in 2016, rested on an increase in dystrophin expression measured in muscle biopsy. The increase was small in absolute terms, and the clinical relevance of that magnitude was disputed within the agency to an unusual and publicly documented degree. A decade on, the dystrophin surrogate remains contested, and the episode is cited both as evidence that the pathway enables access and as evidence that it can be stretched past its evidentiary basis. Tofersen, for SOD1-associated amyotrophic lateral sclerosis, presents a cleaner case. Its pivotal trial did not meet its primary clinical endpoint, but it produced a large, consistent reduction in neurofilament light chain, a marker of axonal injury with a substantial body of evidence linking it to disease progression across neurodegenerative conditions. The FDA granted accelerated approval in 2023 on that basis, with longer-term follow-up suggesting clinical benefit emerging later than the trial's original window. The difference between the two cases is not the pathway; it is the depth of the prior evidence linking the marker to the outcome, and the consistency of the treatment effect on it. The design lesson is that a surrogate's value is established long before the pivotal trial, in natural history cohorts where the marker's relationship to progression can be characterised in untreated patients. A sponsor who begins collecting the biomarker at the start of the pivotal trial has no way to demonstrate that relationship. A sponsor who has been collecting it in the registry for six years has a case. An endpoint built for one disease Voretigene neparvovec, a gene therapy for inherited retinal dystrophy caused by biallelic mutations in a specific gene, illustrates what purpose-built measurement can achieve. The clinical problem was that conventional ophthalmological endpoints — visual acuity on a letter chart, visual field extent — did not capture the disability that mattered to these patients, which was the inability to navigate in dim light. Acuity can be relatively preserved while a patient is functionally blind at dusk. The response was to construct a multi-luminance mobility test: a standardised obstacle course that a patient navigates at several specified illumination levels, scored on accuracy and time, with the outcome expressed as the change in the lowest light level at which the patient can pass. The measure was developed, validated, and agreed with the regulator before the pivotal trial, which then enrolled a few dozen patients and produced a result that was interpretable because the endpoint measured the thing the therapy was supposed to change. Three features of this are generalisable. The endpoint was derived from what patients identified as their limitation rather than from what the specialty conventionally measured. It was constructed to be objective and instrumented — a physical course with defined lighting, not a clinician's judgement — which made it usable in a trial that could not be perfectly masked. And it produced an ordinal outcome with a natural interpretation, so that a change of two light levels meant something concrete rather than requiring a debate about minimal clinically important differences. The cost was several years of development work before the pivotal trial began. For a programme that needed to demonstrate benefit in a few dozen patients, that investment did more for the probability of a successful readout than any conceivable choice of analysis method. Digital measurement The most consequential recent development in rare disease endpoints is the move from episodic clinic assessment to continuous measurement at home. A six-minute walk test samples a patient's function once every six months, in an artificial setting, on a day they travelled to a hospital. A thigh-worn accelerometer samples continuously. The exemplar is stride velocity 95th centile — the speed of a patient's fastest strides, measured by a wearable device during ordinary life — which the EMA qualified as a primary endpoint in ambulatory Duchenne muscular dystrophy trials. The significance is not the specific measure but the precedent: a continuously collected, device-derived, home-based endpoint accepted as primary evidence in a registrational setting. For small trials the statistical attraction is obvious. Continuous sampling reduces measurement error dramatically; it removes the white-coat and encouragement effects; and it captures function in the environment where it matters. The cautions are equally real. Device-derived endpoints have their own failure modes: adherence decays, firmware changes mid-trial, algorithms are proprietary and may be revised, and the quantity derived from raw accelerometry depends on processing choices that must be locked before unblinding as rigorously as any statistical analysis plan. And a wearable measures what the patient does, which is not the same as what they can do — a measure of activity can fall because a patient became depressed or the weather turned. The discipline is the same as for any other endpoint: characterise it in the natural history cohort first, understand its variance components, and pre-specify everything. Choosing under constraint Putting this together, the endpoint decision in a rare disease programme should proceed roughly as follows. Establish the estimand first, as Chapter 2 argued: which population, which variable, and how intercurrent events are handled. Then ask what the natural history cohort shows about candidate measures — how fast each changes, with what variability, in which patients, with what floor and ceiling behaviour. Compute the implied standardised effect for each candidate under a plausible treatment effect, and look at the resulting sample sizes side by side. This exercise routinely reveals that a secondary endpoint has better operating characteristics than the intended primary, and it is far better to discover that before the protocol is written. Then ask what measurement improvements are available and price them. Duplicate baselines, central certification of assessors, blinded video adjudication, and covariate adjustment are collectively capable of the sort of variance reduction described at the start of this chapter, and they cost a fraction of what enrolling twice as many patients would cost — if enrolling twice as many patients were even possible, which is the point. Finally, take the endpoint to the regulator before the design. The endpoint discussion is where the programme's evidentiary foundation is either laid or left to chance, and an agency that has agreed the primary endpoint is a much easier audience for a novel design than one encountering both at once. The next chapter turns to where the comparison comes from — and the natural history cohort that this chapter has repeatedly leaned on turns out to be the single most valuable asset a rare disease programme can own. Hashtags: #ClinicalTrialDesignInRareDiseases #RareDiseaseTrials #SmallSampleTrials #BayesianAdaptiveDesigns #AdaptivePlatforms #BayesianClinicalTrials #NaturalHistoryData #ExternalControlArms #HistoricalControls #BayesianBorrowing #PowerPriors #CommensuratePriors #MetaAnalyticPredictivePriors #RobustPriors #HierarchicalModels #BasketTrials #NOf1Trials #MasterProtocols #MultiArmMultiStageTrials #SharedControlGroups #TrialSimulation #PriorEffectiveSampleSize #RegulatoryFlexibility #RareDiseaseEndpoints #FutureOfRareDiseaseTrials

Download the Book (PDF): Introduction A molecular assay that works in a research laboratory has proved one thing: that under favorable conditions, in the hands of the people who built it, it can detect or measure what it was designed to detect or measure. That is a genuine achievement, and it is usually the product of months or years of primer design, optimization, and troubleshooting. It is also not nearly enough to put a result in a patient's chart. The gap between those two states is the subject of this book. On one side sits the bench assay: run in batches when a graduate student has time, calibrated against whatever standard was convenient, tested on archived specimens that were chosen because they were interesting, and judged by whether the figures in a manuscript look convincing. On the other side sits the clinical test: run every day by rotating staff, on specimens that arrive hemolyzed, lipemic, under-filled, or delayed in transit, reported to physicians who will change a dose, start a therapy, or reassure a patient on the strength of a single number. The clinical test must behave the same way on a Tuesday night in February as it did on the afternoon it was validated, and the laboratory must be able to prove that it does. The proof is what regulators and accreditors mean by validation. In the United States, the Clinical Laboratory Improvement Amendments of 1988 (CLIA) and their implementing regulations at 42 CFR Part 493 require any laboratory that tests human specimens for clinical purposes to hold a certificate and to meet quality standards. The central provision for anyone bringing a new test into service is 42 CFR 493.1253, which draws a sharp line. A laboratory that adopts an unmodified test cleared or approved by the Food and Drug Administration must verify that it can reproduce the manufacturer's performance for accuracy, precision, and reportable range, and that the manufacturer's reference intervals suit its patients. A laboratory that modifies such a test, or introduces a test of its own design, must establish performance specifications for accuracy, precision, analytical sensitivity, analytical specificity including interfering substances, reportable range, reference intervals, and any other characteristic the test requires. An assay transferred from a university bench almost always falls on the second side of that line. It is a laboratory-developed test, and the laboratory carries the full burden of establishing what it can and cannot do. Accreditation adds a second layer. Most hospital and academic laboratories that perform high-complexity molecular testing are accredited by the College of American Pathologists (CAP), whose checklists translate the regulatory requirements into specific, inspectable expectations: a validation study reviewed and signed by the laboratory director before patient testing begins, an inventory of laboratory-developed and modified tests, a statement on the report that the test was developed by the laboratory, validation of any clinical claims the laboratory makes, and documented plans for handling known interferences. Laboratories outside the United States, and many within it, also work to ISO 15189, the international standard for medical laboratory quality and competence, now in its 2022 edition. Tests performed on specimens from New York State residents must additionally be approved by that state's Clinical Laboratory Evaluation Program. For several years the landscape looked as though it might change fundamentally. In May 2024 the FDA published a final rule declaring that laboratory-developed tests were medical devices under the Federal Food, Drug, and Cosmetic Act and announcing a four-year phaseout of its long-standing policy of enforcement discretion. Laboratories began planning for premarket submissions, quality system requirements, and adverse event reporting. On 31 March 2025, the United States District Court for the Eastern District of Texas vacated that rule in litigation brought by the American Clinical Laboratory Association and the Association for Molecular Pathology, holding that laboratory-developed tests are professional services rather than manufactured devices and fall outside the FDA's statutory authority. The government did not appeal, and in September 2025 the FDA published a notice in the Federal Register removing the rule's text from the Code of Federal Regulations. As of this writing in 2026, the analytical quality of laboratory-developed tests in the United States is governed by CLIA, by accreditation standards, and by state programs such as New York's. Congress continues to consider legislation, and the Centers for Medicare & Medicaid Services issued a request for information in July 2026 on modernizing the CLIA regulations, including performance standards for laboratory-developed tests. None of that has yet changed what a laboratory must do today. That regulatory history matters for a practical reason. The vacatur did not lower the bar for validation; it confirmed where the bar sits. The legal argument that prevailed was that clinical laboratories practice a professional service already overseen by CLIA. The strength of that position depends on laboratories actually doing what CLIA and their accreditors require, and doing it well. A laboratory that treats validation as paperwork weakens the case that the current framework is adequate. A laboratory that treats validation as science strengthens it. The argument of this book is that validation is best understood as an evidentiary argument rather than a checklist. Every performance characteristic named in the regulation is a question about error: how small a quantity can the test distinguish from nothing, how often does it mistake something else for the target, how much do its results scatter when nothing about the specimen has changed, how far do its results sit from the truth, how does the specimen itself distort the measurement, and what range of results should a clinician regard as unremarkable. A good validation answers each of those questions with data collected under realistic conditions, sets the answers against performance goals chosen before the data were collected, and draws a conclusion a skeptical reviewer would accept. The checklist tells you which questions will be asked. It does not tell you whether your answers are good enough, and it cannot tell you what your particular assay's weak points are. Only experimental design grounded in an understanding of the method can do that. The book follows that logic. Chapter 1 maps the regulatory landscape in enough detail to know which requirements apply to which kind of test, including where the FDA's position now stands. Chapter 2 treats validation as study design: defining intended use, fixing the method before the clock starts, choosing performance goals, and sourcing samples. Chapters 3 through 8 take the performance characteristics in turn. Analytical sensitivity covers the limits of blank, detection, and quantitation and the probit and hit-rate approaches that dominate molecular work. Analytical specificity covers interference, cross-reactivity, inclusivity, and the contamination risks peculiar to amplification methods. Precision covers the design and analysis of replication experiments and how to read variance components. Accuracy and reportable range cover method comparison, linearity, and dilution. Matrix effects cover the ways the specimen type, collection tube, extraction, and stability can bend results, and the problem of commutability in reference materials. Reference intervals cover how to establish, transfer, and verify the ranges and decision limits that give a result clinical meaning. Chapter 9 turns the finished validation into a running service: quality control design, proficiency testing and its alternatives, director approval, reporting, change control, and the inspection. The conclusion argues for what follows from treating validation as evidence rather than ritual. Throughout, the book leans on the consensus documents of the Clinical and Laboratory Standards Institute (CLSI), which supply the experimental designs that inspectors expect to see and that the FDA recognizes as consensus standards. The ones that recur are EP05 for precision, EP06 for linearity, EP07 for interference, EP09 for method comparison, EP12 for qualitative tests with binary output, EP15 for user verification of precision and bias, EP17 for detection capability, and EP28 for reference intervals, supported by others on specimen equivalence, commutability, and multiplex nucleic acid assays. These documents are not law, and a laboratory may use other defensible designs. But they encode decades of statistical thinking about how to extract honest estimates from limited experiments, and departing from them without good reason invites questions that are hard to answer. Worked examples appear in most chapters. Where they involve a specific assay, the assay is hypothetical, most often a laboratory-developed quantitative PCR test for BK polyomavirus DNA in plasma and urine, chosen because it exercises nearly every characteristic the regulation names: a detection limit that matters clinically, a quantitative range spanning several orders of magnitude, two specimen types with very different matrices, genotypic diversity in the target, and a need for results that are comparable over time as a patient is monitored. The numbers in those examples are illustrative. They show how the calculations work and how the judgments are made; they are not claims about any real assay's performance. The reader this book has in mind is a scientist or physician who has built something that works and now needs to make it clinical: a principal investigator whose assay has attracted clinical interest, a laboratory director asked to bring a research method into service, a molecular technologist writing the validation plan, a quality manager preparing for inspection, or a trainee in clinical chemistry, microbiology, or molecular pathology. The aim is not to reproduce the CLSI documents, which should be read in the original, but to explain what each experiment is for, how its results should be judged, and how the pieces fit together into a case that stands up. Chapter 1: The Regulatory Landscape for Clinical Tests A research assay becomes a clinical test at the moment a result derived from it is used for the diagnosis, prevention, or treatment of disease, or for the assessment of a person's health. That functional definition, written into CLIA, is what brings a laboratory under federal oversight. It does not matter whether the laboratory calls itself a research core, whether the assay was published, or whether the physician ordering it is also a co-investigator. If individual results go back to a patient or into a medical record to guide care, the testing must be performed in a CLIA-certified laboratory under CLIA's quality standards. Research testing whose results are never reported to individuals for clinical purposes stays outside CLIA, and the moment that boundary is crossed is the moment the obligations in this book begin. CLIA: certificates, complexity, and the core obligation CLIA is administered by the Centers for Medicare & Medicaid Services (CMS), with the Centers for Disease Control and Prevention providing scientific support and the FDA responsible for categorizing commercially marketed tests by complexity. The regulations sort tests into three tiers: waived, moderate complexity, and high complexity. Waived tests are simple enough that the risk of an erroneous result is low; a laboratory performing only waived tests needs a certificate of waiver and must follow the manufacturer's instructions, but it is not subject to the performance-specification requirements discussed here. Moderate- and high-complexity tests are nonwaived, and a laboratory performing them needs a certificate of compliance, issued after inspection by a state agency on behalf of CMS, or a certificate of accreditation, issued when the laboratory is accredited by a CMS-approved accrediting organization with deemed status. CAP, COLA, The Joint Commission, AABB, A2LA, and ASHI hold that status for their respective scopes. Laboratory-developed tests are not categorized by the FDA, and CMS treats them, along with modified FDA-cleared or approved tests, as high complexity. That classification carries consequences beyond validation. High-complexity testing requires a laboratory director with specified qualifications, a technical supervisor, a clinical consultant, general supervisors, and testing personnel who meet education and training standards. Those personnel standards were revised in a CMS final rule published on 28 December 2023 that took effect on 28 December 2024, the first substantial update in three decades. The revisions adjusted degree equivalency pathways and clarified director responsibilities, including on-site visits. A university research group transferring an assay should confirm early that the people who will perform and supervise clinical testing meet these standards, because a technically sound validation performed by staff who do not qualify to run the test is a problem no statistical analysis can solve. The analytic core of CLIA sits in Subpart K of Part 493, which covers quality systems for nonwaived testing. The section that governs bringing a new method into service is 493.1253, "Establishment and verification of performance specifications." Its structure is short and worth knowing by heart. Paragraph (b)(1) applies to a laboratory introducing an unmodified, FDA-cleared or approved test system: before reporting patient results, the laboratory must demonstrate that it can obtain performance comparable to the manufacturer's specifications for accuracy, precision, and reportable range, and must verify that the manufacturer's reference intervals are appropriate for its patient population. Paragraph (b)(2) applies to a laboratory that modifies an FDA-cleared or approved test, introduces a test system not subject to FDA clearance or approval, or uses a test system whose performance specifications are not provided by the manufacturer. Such a laboratory must, before reporting, establish performance specifications for accuracy, precision, analytical sensitivity, analytical specificity to include interfering substances, reportable range, reference intervals, and any other performance characteristic required for test performance. Paragraph (b)(3) requires the laboratory to determine its calibration and control procedures based on the specifications verified or established. Paragraph (c) requires documentation of all of it. Two features of that text shape everything that follows. The first is the verb. Verification is confirmation: the manufacturer has already done the heavy experimental work, submitted it to the FDA, and published claims, and the laboratory shows it can reproduce them in its own hands. Establishment is discovery: nobody has characterized the test before, and the laboratory must generate the primary evidence. The second is the phrase "any other performance characteristic required for test performance." The regulation lists six characteristics, but it does not pretend the list is exhaustive. For a molecular assay, carryover contamination, extraction efficiency, specimen stability, and inclusivity across genotypes are all performance characteristics required for the test to perform, and an inspector is entitled to ask about them. Table 1 sets the two paths side by side. Table 1. Verification and establishment under 42 CFR 493.1253. Characteristic Unmodified FDA-cleared test Modified FDA test or LDT Typical CLSI reference Accuracy Verify Establish EP09, EP15 Precision Verify Establish EP05, EP15 Reportable range Verify Establish EP06, EP34 Reference intervals Verify suitability Establish EP28 Analytical sensitivity Not required by rule Establish EP17, EP12 Analytical specificity and interference Not required by rule Establish EP07, EP37 The practical upshot for an assay moving from a university bench is that nearly every row reads "establish." There is no manufacturer's package insert to lean on and no FDA review behind the method. Even when a laboratory builds its test around commercially available components, such as an FDA-cleared extraction platform or analyte-specific reagents sold for clinical use, the assembled test is a laboratory-developed test and the laboratory owns its performance. What counts as a modification is a frequent source of trouble. Using a cleared test on a specimen type not listed in its instructions for use, changing an incubation time, altering a cutoff, substituting a different extraction method, or applying the test to a population outside its intended use all turn a verification into a validation. The CAP checklists are explicit that such changes convert a cleared test into a modified one requiring full establishment of performance specifications. A laboratory that runs an FDA-cleared respiratory panel on bronchoalveolar lavage fluid when the claim covers only nasopharyngeal swabs has, for that specimen type, a laboratory-developed test. Accreditation: what CAP adds CLIA sets the floor. Accreditation organizations must impose requirements at least as stringent, and CAP's checklists go into considerably more detail. For a laboratory accredited by CAP, the relevant requirements for validation live mainly in the All Common Checklist, which applies to every laboratory section, and in discipline-specific checklists such as Molecular Pathology, Microbiology, and Chemistry and Toxicology. The checklists are revised annually and their item numbers change from time to time, so a validation plan should cite the current edition rather than a number remembered from an old inspection. The concepts, however, have been stable for years. First, the laboratory director or a qualified designee must review the validation or verification data and sign a written statement that the performance is acceptable before the test is used for patient care. That signature is not ceremonial. It is the point at which a physician or doctoral scientist with legal responsibility for the laboratory's results accepts that the evidence supports clinical use. A validation summary that presents data without a clear conclusion, or that presents a conclusion unsupported by the data, puts the director in the position of signing something indefensible. Second, for laboratory-developed and modified tests the checklist expects the evaluation to include analytical sensitivity, analytical specificity including interferences, and validation of any clinical claims the laboratory makes. For accuracy studies on these tests, the checklist has for several years set a default expectation of at least 20 samples, spanning the measuring interval for quantitative tests and including positive, negative, and low-positive samples for qualitative ones, with a documented justification from the director if fewer are used. Twenty is a floor, not a design target, and for most molecular assays with meaningful claims the defensible number is larger. Third, the laboratory must keep a list of its laboratory-developed and modified tests, and patient reports for laboratory-developed tests must carry a statement that the test was developed and its performance characteristics determined by the laboratory. A related disclaimer is required by FDA regulation at 21 CFR 809.30 when a test is built on analyte-specific reagents, and many laboratories use a single statement that satisfies both. Fourth, the checklist asks the laboratory to know the interferences that affect each test and to have a written plan for what to do when a specimen is affected, to make its performance specifications available to clients on request, and to notify clients when a methodology change produces results that differ significantly from before. The Molecular Pathology Checklist layers on requirements specific to nucleic acid testing: physical separation or other controls against amplicon contamination, monitoring for contamination, appropriate controls for extraction and amplification including inhibition, documentation of primer and probe sequences or their equivalent, and, for sequencing-based tests, requirements for bioinformatic pipelines and their validation. CAP inspections occur every two years, with a self-inspection in the intervening year, and inspectors are practicing laboratory professionals who will read the validation packet and ask the people who performed it to explain it. ISO 15189:2022 frames the same obligations in the language of an international quality management standard. It distinguishes verification of validated methods from validation of non-standard or laboratory-designed methods, requires that validation be as extensive as necessary to confirm fitness for intended use, and places explicit emphasis on evaluating measurement uncertainty and on risk management. Laboratories accredited to ISO 15189 in the United States typically also hold CLIA certificates, and a well-designed validation can satisfy both frameworks with the same data. The FDA and laboratory-developed tests, 2024 to 2026 For most of CLIA's history, the FDA took the position that it had authority to regulate laboratory-developed tests as medical devices but chose not to exercise it, a posture described as enforcement discretion. That discretion never extended to the components laboratories buy: instruments, general-purpose reagents, analyte-specific reagents, and software marketed as devices remained under FDA jurisdiction. Nor did it extend to kits sold to other laboratories, which require clearance, approval, or authorization. On 6 May 2024, the FDA published a final rule, "Medical Devices; Laboratory Developed Tests," at 89 Federal Register 37286. The rule amended the definition of in vitro diagnostic products in 21 CFR 809.3 to state that such products include those whose manufacturer is a laboratory, and it laid out a phased end to enforcement discretion over four years: medical device reporting and complaint handling first, then registration and listing and labeling, then quality system requirements, and finally premarket review for high- and moderate-risk tests. The first stage was scheduled for May 2025. The American Clinical Laboratory Association and the Association for Molecular Pathology sued in the Eastern District of Texas, and the cases were consolidated. On 31 March 2025, the court granted summary judgment for the plaintiffs and vacated the rule. The court held that laboratory-developed tests are professional services performed by laboratories rather than devices introduced into commerce, that the statutory definition of a device does not reach them, and that Congress had addressed laboratory testing through CLIA. The decision drew heavily on the Supreme Court's 2024 ruling in Loper Bright Enterprises v. Raimondo, which ended judicial deference to agency interpretations of ambiguous statutes. The government did not appeal. On 19 September 2025, the FDA published a notice implementing the vacatur and restoring the prior text of 21 CFR 809.3, and the agency's web pages on laboratory-developed tests now describe the rule as vacated. Several points follow for a laboratory planning a validation in 2026. The requirements that govern the analytical validity of a laboratory-developed test are those of CLIA, the laboratory's accreditor, and any applicable state program. The FDA continues to regulate the products laboratories purchase and continues to require premarket review of test kits sold to other laboratories, so a university group that intends to commercialize its assay as a distributed kit rather than offer it as a single-site service is on a different path entirely, governed by device law and design controls. Companion diagnostics raise their own questions, because a therapeutic's labeling may specify an FDA-approved test. And the policy debate is live. Legislation to create a new framework for diagnostics, in various forms under names such as the VALID Act, has been introduced repeatedly without passage, and in May 2026 a bill was introduced in the House that would affirm that laboratory-developed tests are not devices and would expand CMS authority over them under CLIA. In July 2026 CMS published a request for information on modernizing the CLIA regulations, asking among other things about performance standards for laboratory-developed tests, particularly next-generation sequencing, about verification of software and artificial intelligence tools, and about laboratories that interpret data without handling specimens. The comment period closed in September 2026. None of these initiatives had changed the regulations at the time of writing, but they indicate the direction of attention: toward more specific expectations for how laboratory-developed tests are validated, not fewer. A laboratory that validates its tests rigorously loses nothing whichever way that debate resolves. If the framework remains CLIA-centered, rigorous validation is what CLIA and CAP already require. If a future framework adds premarket review for some categories of test, the evidence generated under a sound CLIA validation is the raw material a submission would need. Other layers: states, payers, and professional guidelines New York and Washington are the two states whose laboratory programs CMS has found equivalent to CLIA, exempting laboratories licensed there from federal CLIA certification. New York's Clinical Laboratory Evaluation Program, run through the Wadsworth Center, has a particular reach: any laboratory, wherever located, that tests specimens from New York residents must hold a New York permit, and laboratory-developed tests must be approved by the program before use on those specimens. The program publishes submission guidelines for specific test categories, including molecular infectious disease assays and oncology panels, and its reviewers read validation packages closely. A laboratory that expects to receive New York specimens should read those guidelines while designing its validation, not after, because retrofitting a missing experiment is much harder than planning it. Payers exert a softer form of pressure. Coverage decisions for molecular tests, particularly under the MolDX program that Palmetto GBA developed and several Medicare administrative contractors have adopted, may require a technical assessment that asks for evidence of analytical and clinical validity. The standards are not identical to CLIA's, and they generally ask more about clinical validity and utility, but a clean analytical validation is a prerequisite. Finally, professional bodies publish recommendations that inspectors and reviewers treat as a benchmark of reasonable practice. The Association for Molecular Pathology and CAP have issued joint guidelines on validating next-generation sequencing oncology panels and bioinformatics pipelines; the American College of Medical Genetics and Genomics publishes standards for germline testing; and the Clinical and Laboratory Standards Institute maintains the evaluation protocols this book relies on. None of these is binding by itself, but a design that ignores them without explanation is hard to defend. Where the burden actually lies It is tempting, after reviewing all of this, to see validation as an exercise in compliance with a stack of overlapping authorities. That reading misses the point of the regulation's language. Section 493.1253 does not tell a laboratory how many replicates to run or what acceptance criteria to use. It tells the laboratory to establish performance specifications, which means determining, with evidence, how the test performs. The judgment about whether that performance is good enough rests with the laboratory director, informed by the intended clinical use. That design places an unusual degree of trust in the laboratory. It assumes that the people who develop and run a test understand its failure modes better than any external reviewer could and will design experiments that expose them. The rest of this book is about honoring that trust: designing a validation that asks the hard questions of the assay, and assembling an answer that a knowledgeable skeptic would accept. Chapter 2: Designing the Validation Before Running It The most common reason a validation fails inspection is not that the assay performed badly. It is that nobody can say, from the documents, what the assay was supposed to do, what performance would have been acceptable, or why the experiments that were run are sufficient to show it. The data exist, often in abundance, but they were collected without a plan and interpreted after the fact. A validation designed that way can be made to support almost any conclusion, which is exactly why a reviewer will not trust it. The remedy is to treat validation as a prospective study. Before the first validation run, the laboratory writes down the intended use of the test, freezes the procedure, states the performance goals each experiment must meet, specifies the experiments and samples, and defines how the data will be analyzed. The director approves that plan. Then the laboratory executes it, records deviations as deviations, and writes a report that compares results with the pre-stated goals. This is the same logic that governs clinical trials, and for the same reason: it separates what the investigators hoped to see from what the data show. Intended use as the first design decision Every downstream choice depends on a precise statement of what the test is for. An intended use statement names the analyte, the specimen types, the population, the clinical purpose, and the form of the result. Consider the hypothetical BK polyomavirus assay that recurs in this book. A research group has developed a quantitative PCR targeting a conserved region of the viral genome, has used it to study viral kinetics in kidney transplant recipients, and now wants to offer it clinically. Its intended use might read: "Quantitative detection of BK polyomavirus DNA in EDTA plasma and in urine from kidney transplant recipients, as an aid in monitoring for BK virus reactivation and nephropathy and in managing immunosuppression. Results are reported in international units per milliliter." Each clause commits the laboratory to evidence. "Quantitative" means the validation must establish a lower limit of quantitation, a linear range, and precision across that range, not just a detection limit. "EDTA plasma and urine" means two matrices with very different composition, each of which needs its own characterization of sensitivity, inhibition, and stability; a result established in plasma does not transfer to urine by assumption. "Kidney transplant recipients" defines the population in which clinical performance and any reference or decision values are relevant. "Monitoring" implies that clinicians will compare successive results in the same patient, so between-run precision and long-term stability of calibration matter as much as accuracy against any single comparator. "International units" commits the laboratory to calibration traceable to the World Health Organization International Standard for BK virus DNA, with all the questions about commutability that raises. A vaguer intended use, such as "detection of BK virus," would seem to lighten the load but in practice creates problems. Clinicians will use the test in ways the laboratory never characterized: on cerebrospinal fluid, on whole blood, in stem cell transplant recipients, to follow trends. When a result misleads, the laboratory will have no evidence that the test was fit for the purpose to which it was put. Precision in the intended use protects both patients and the laboratory, and it defines the boundary beyond which any new use becomes a modification requiring its own validation. Freezing the method A research assay is a moving target. Primer concentrations are adjusted, a new master mix is tried, the cycling program is shortened, the extraction kit is changed because the old one is back-ordered. That flexibility is the point of development. It is incompatible with validation, which characterizes a specific, fixed procedure. Data collected on one version of the method say little about another. The practical step is to write the clinical standard operating procedure before validation begins and to treat it as locked for the duration. The procedure specifies every reagent by manufacturer and catalog number, every instrument by model, every volume, incubation, and cycling parameter, the calibration scheme, the control materials, the data analysis settings such as baseline and threshold rules, and the criteria for accepting a run and interpreting a result. If a flaw emerges during validation that requires changing the procedure, the change is documented, the director decides which experiments must be repeated, and the data collected under the old version are set aside or explicitly justified as still applicable. Freezing the method forces decisions that research groups often defer. Reagents labeled "for research use only" were not manufactured under quality systems that guarantee lot-to-lot consistency, and a clinical laboratory that builds a test around them takes on the job of qualifying each lot. Where equivalent reagents manufactured for clinical use exist, including analyte-specific reagents, switching to them before validation is usually worth the effort. Oligonucleotides should come from a supplier that provides documentation of synthesis and purification, and the laboratory should decide how it will accept each new synthesis lot. Custom calibrators prepared in the research laboratory need a documented preparation procedure, value assignment, and stability data. Software used to call results, whether an instrument's built-in analysis or a script written by a postdoctoral fellow, is part of the method and must be version-controlled and validated with the rest. This is also the point at which to settle the question of feasibility. A short series of preliminary experiments, sometimes called a pre-validation or feasibility phase, lets the laboratory confirm that the frozen method behaves roughly as expected in the clinical setting, on the clinical instruments, with clinical staff. CLSI once published a document, EP10, specifically for a quick preliminary evaluation of this kind. The data from feasibility work are useful for setting realistic goals and sample sizes, but they are not validation data, and mixing the two erodes the prospective character of the study. Performance goals and where they come from A performance goal states, before the experiment, the result that would make the assay acceptable for its intended use. "The coefficient of variation for between-day precision will not exceed 25 percent at 1,000 IU/mL" is a goal. "Precision will be determined" is not. Without goals, the validation report can only describe the assay; with them, it can conclude something about it. Setting goals is the hardest intellectual work in validation, because it requires connecting analytical error to clinical consequence. The field's most widely cited framework for doing so is the consensus reached at a 2014 conference in Milan organized by the European Federation of Clinical Chemistry and Laboratory Medicine, which recognized three bases for analytical performance specifications. The first is the effect of analytical performance on clinical outcomes, which is the most relevant but rarely available, because it requires studies showing how measurement error alters clinical decisions and patient results. The second is biological variation: the principle that analytical imprecision and bias should be small relative to the natural within- and between-person variability of the quantity measured, so that the test adds little noise to the signal. The third is the state of the art: the performance that the best available methods achieve, as shown for example by proficiency testing data. The Milan consensus did not rank these as interchangeable; it suggested choosing the model that best fits the analyte and its clinical use. For molecular tests, the biological variation model often fits awkwardly. Viral load in a transplant patient is not a homeostatic quantity fluctuating around a set point; it can change by orders of magnitude over weeks. What matters clinically is usually whether a change between two measurements reflects a real change in the patient. For quantitative nucleic acid tests, a long-standing rule of thumb holds that differences smaller than about half a log10 unit, roughly threefold, should not be interpreted as biologically meaningful, because typical assay variability approaches that magnitude near the low end of the range. A laboratory that wants its assay to detect a threefold change reliably must therefore show that its total variability, including between-run, between-operator, and between-lot components, is comfortably smaller than that. The clinical decision points supply another anchor. If transplant physicians act when plasma BK DNA exceeds a particular threshold, then precision and accuracy near that threshold matter more than performance elsewhere, and the validation should concentrate samples there. Guidelines from transplant societies have discussed plasma BK DNA thresholds associated with presumptive nephropathy, but those thresholds were derived with specific assays, and the absence of complete harmonization among BK assays means a laboratory cannot assume that a published threshold applies to its own numbers without evidence of agreement. That caution recurs in Chapter 8. For detection limits, goals typically come from clinical need and from comparison with alternatives. A test intended to rule out early reactivation must detect DNA at concentrations where clinical intervention is still useful; a test whose detection limit sits well above that of the method clinicians have been using will produce discordant, confusing results. For specificity, the goal is usually framed as the absence of false positives from a defined panel of cross-reactants and interferents under stated conditions, and the laboratory decides in advance what effect size counts as interference. Where a regulatory standard exists, it sets a minimum. CLIA's proficiency testing criteria define acceptable performance for regulated analytes, and a 2022 final rule revised those criteria and added new analytes, with the changes taking effect in July 2024 and applying to proficiency testing events from 2025. Most molecular analytes are not regulated analytes under CLIA, so those criteria rarely apply directly, but where they do they are a floor, not a goal; an assay that barely meets proficiency testing limits is one bad day away from failure. The validation plan With intended use, method, and goals settled, the plan specifies the experiments. A complete plan for a quantitative molecular laboratory-developed test typically addresses the following, in roughly the order they are run: Detection capability: limit of blank, limit of detection, and lower limit of quantitation, in each specimen type, using the design from Chapter 3. Analytical specificity: cross-reactivity with related organisms and common flora, inclusivity across target variants, and interference from endogenous and exogenous substances, including inhibition. Precision: repeatability and within-laboratory precision at several concentrations, including one near the lower limit of quantitation and one near each clinical decision point. Accuracy: comparison with a reference or established method on patient specimens, recovery of reference materials, and agreement with external quality assessment samples where available. Reportable range: linearity across the measuring interval and, if dilutions will be used to extend it, validation of the dilution protocol. Matrix and specimen studies: equivalence of specimen types and collection tubes, stability under realistic storage and transport conditions, and carryover. Reference intervals or clinical decision values, as appropriate to the analyte. Any other characteristic the method demands, such as the performance of an internal control or the accuracy of a bioinformatic call. For each experiment the plan states the samples, number of replicates, days, runs, operators, reagent lots, and instruments; the statistical analysis; and the acceptance criterion. It also states what happens if a criterion is missed: whether the experiment may be repeated, under what conditions, and whether a miss can be accepted with a documented limitation. Two design choices deserve special attention because they are often underplanned. The first is the representation of real sources of variation. A precision experiment run by one person, on one instrument, with one lot of reagents, over three days produces an estimate that will be too optimistic for a clinical service that runs every day on two instruments with several technologists and a new lot of master mix every few months. The plan should ensure that the factors that will vary in practice vary in the validation. At a minimum, that means more than one operator, more than one reagent lot where feasible, every instrument that will be used clinically, and enough days to capture day-to-day change. The second is sample size. The number of samples determines the precision of the estimates, and an estimate with a wide confidence interval can meet a goal by luck. Where a claim is framed as "at least 95 percent," the plan should consider what lower confidence bound the data will support, not just the point estimate. What a plan entry looks like Abstract advice about plans is easy to accept and hard to apply, so it helps to see one entry written out. For the hypothetical BK assay, the precision section of the plan might read, in substance, as follows. Objective: to establish repeatability and within-laboratory precision of the whole procedure, from extraction to result, in EDTA plasma and in urine. Samples: for each specimen type, four pools prepared from negative native matrix spiked with whole virus calibrated to the WHO International Standard, at nominal concentrations of 3.0, 4.0, 5.5, and 7.0 log10 IU/mL, the lowest chosen to sit near the expected lower limit of quantitation and the second near the plasma decision point clinicians use. Pools are aliquoted and frozen at −70 °C, with one aliquot thawed per run. Design: two runs per day, two independent extractions per sample per run, for 20 operating days; at least three technologists, each performing no fewer than five days; both extraction instruments and both thermocyclers in rotation; a second lot of extraction reagent introduced on day 11. Analysis: log10-transformed results analyzed by nested random-effects analysis of variance, reporting repeatability, between-run, between-day, and within-laboratory standard deviations with 95 percent confidence intervals, and the lot effect estimated as a fixed factor. Acceptance: within-laboratory standard deviation no greater than 0.25 log10 at 3.0 log10 IU/mL and no greater than 0.17 log10 at 4.0 log10 IU/mL and above, judged on the point estimate, with the upper confidence limit reported; a difference between reagent lots of no more than 0.2 log10. Deviations: a run rejected for a documented technical failure may be repeated on an additional day; no more than two such repeats per sample without director review. Outliers: investigated for assignable cause; excluded only with a documented cause. An entry like that takes a page. It forces every decision that would otherwise be made informally, it tells the technologists exactly what to do, and it gives the reviewer a fixed standard against which to read the results. It also exposes weaknesses before any work is done. Writing "at least three technologists" reveals that the research laboratory has only one person trained on the assay; writing "second lot on day 11" reveals that the laboratory has not yet ordered it. It is far cheaper to discover such gaps in a document than in the middle of a 20-day study. Sourcing samples A validation is only as convincing as its samples. Molecular assays draw on four kinds, each with strengths and weaknesses. Native clinical specimens are the most realistic, because they contain the target in its natural form, in the right matrix, with the right mix of interferents. Residual specimens left over from routine testing are the usual source. Institutional policies on using them vary; many institutions treat the use of de-identified residual specimens for validating a clinical test as quality improvement rather than human subjects research, but the laboratory should confirm the local position with its institutional review board and document it. Native positive specimens at low concentrations, and specimens containing uncommon variants or cross-reactants, may be scarce, and the plan must be realistic about what can be obtained. Contrived specimens are made by spiking a characterized target into negative native matrix: cultured virus, quantified plasmid, synthetic DNA, or a characterized positive specimen diluted into pooled negative material. Contrived specimens make it possible to test concentrations and variants that native specimens cannot supply and to control the true value. Their weakness is that they may not behave like natural infection. Plasmid DNA is not encapsidated, so it bypasses the lysis step and can overstate extraction efficiency; cultured virus may differ in the proportion of intact virions from virus shed in a patient. The plan should state which experiments use contrived material and justify that the material is suitable. Reference materials with assigned values, including WHO International Standards and secondary standards calibrated against them, anchor quantitative results to a common scale. They are indispensable for calibration and for establishing traceability, but they come with the commutability problem discussed in Chapter 7: a reference material may not respond to the assay in the same way as patient specimens. External quality assessment and proficiency testing materials from previous surveys, where the laboratory can obtain them, provide an independent check against the values other laboratories obtained. They are useful for accuracy and for anticipating how the assay will fare in future surveys. A mature validation uses all four, each where it is strongest: native specimens for method comparison and for representing real matrices, contrived specimens for detection limits and inclusivity, reference materials for calibration and trueness, and external samples for independent confirmation. The document trail The validation plan, the raw data, the analysis, and the report together form the record that the regulation demands and that the inspector will read. The report should restate the intended use, summarize each experiment against its pre-stated goal, list every deviation and its disposition, state any limitations that will appear in the procedure or on the report, and end with a clear conclusion that the director signs. Raw data should be retained in a form that allows the analysis to be reconstructed, including instrument files, not just transcribed values. Laboratories often underestimate how much the quality of this record affects how an inspector judges the underlying science. A clear record invites confidence; a confusing one invites the inspector to probe until something breaks. Chapter 3: Analytical Sensitivity and the Limits of Detection "Sensitivity" is one of the most overloaded words in laboratory medicine. Clinicians use it to mean the proportion of people with a disease whom a test correctly identifies as positive, a property of the test in a population. Analytical chemists sometimes use it to mean the slope of the calibration curve, the change in signal per unit change in concentration. CLIA uses "analytical sensitivity" to mean the smallest amount of the analyte the method can reliably detect. These are different quantities, and confusing them produces validation reports that answer the wrong question. This chapter concerns the third meaning, which the CLSI guideline EP17 formalizes through three linked concepts: the limit of blank, the limit of detection, and the limit of quantitation. Blank, detection, and quantitation The limit of blank (LoB) is the highest measurement result likely to be observed when a sample containing no analyte is tested. It is a statement about the noise floor. If blank samples produce a spread of small nonzero readings, the LoB is set at a high percentile of that spread, conventionally the 95th, so that a blank will exceed it only about 5 percent of the time. That fixes the rate of false positives from pure noise, the type I error. The limit of detection (LoD) is the lowest amount of analyte that the method will detect with a stated probability, conventionally 95 percent, meaning a sample at that concentration will produce a result above the LoB in 95 percent of tests. It fixes the rate of false negatives at that concentration, the type II error. The LoD is necessarily higher than the LoB, because a sample whose true concentration sits exactly at the LoB would exceed it only about half the time. The limit of quantitation (LoQ) is the lowest amount of analyte that can be measured with acceptable accuracy, where "acceptable" is defined by a goal the laboratory sets for total error, or for imprecision and bias separately. The LoQ can equal the LoD but is often higher, because a signal that is reliably present may still be too noisy to assign a trustworthy number. For a quantitative assay, the lower limit of the reportable range is the LoQ, not the LoD. For a classical continuous measurement, such as an immunoassay reading light units, EP17 gives parametric formulas. With blank measurements approximately normal, LoB equals the mean of the blanks plus 1.645 times their standard deviation. The LoD is then LoB plus 1.645 times the standard deviation of a low-concentration sample, with a small correction factor for the number of measurements. Where blank results are not normally distributed, as when many blanks read exactly zero, EP17 describes a nonparametric approach that takes the 95th percentile of ranked blank results. The guideline recommends that those establishing detection claims use multiple reagent lots, instruments, and days, and on the order of 60 replicates per reagent lot for each of the blank and low-level studies, while laboratories verifying a manufacturer's claim can use a smaller design with about 20 replicates. Table 2 summarizes the terms and how each is typically determined for a molecular assay. Table 2. Detection capability terms and their determination in molecular assays. Term Question answered Typical molecular approach Limit of blank How high does noise read? Negative matrix replicates; often no amplification Limit of detection What is detected 95% of the time? Hit rate by probit over a dilution series C5 to C95 interval Where is the transition from miss to hit? Hit rates bracketing the cutoff (EP12) Limit of quantitation What can be measured to goal? Total error at low levels vs. predefined goal Why molecular assays need a different approach Real-time PCR does not produce a continuous signal at the low end in the way an immunoassay does. A reaction either amplifies to cross the threshold or it does not. A reaction with no template usually produces no cycle threshold at all, reported as "undetermined," and in a well-controlled assay essentially all blanks behave this way. The LoB is therefore effectively zero, and the question of sensitivity becomes a question of probability: at what concentration does the proportion of replicates that amplify reach 95 percent? Two facts shape the answer. The first is sampling statistics. At low concentrations, the number of target molecules that end up in any given reaction follows a Poisson distribution. If the average number of copies per reaction is λ, the probability that a reaction receives no copies at all is e^−λ, and so the best possible detection rate, for a perfect assay that amplifies every single molecule, is 1 − e^−λ. Setting that equal to 0.95 gives λ of about 3. No PCR assay can have a 95 percent limit of detection below about three copies per reaction, whatever its chemistry. An assay that claims otherwise has miscounted its copies. The second fact is that for a clinical molecular assay, the relevant limit is a property of the whole process, from specimen to result, not of the amplification alone. Nucleic acid must be released from the organism or cell, recovered through extraction with some efficiency, eluted into a volume from which only a fraction enters the reaction, and amplified in the presence of whatever inhibitors survived purification. Each step loses or dilutes target. An LoD determined by spiking purified DNA directly into the PCR tells the laboratory about its primers and cycling. It says little about what the test can detect in a patient's plasma. The arithmetic linking the two is simple and worth writing into the validation report. Suppose the hypothetical BK assay extracts 0.4 mL of plasma, elutes into 100 µL, and adds 10 µL of eluate to each PCR. Each reaction then contains the nucleic acid from 0.04 mL of plasma, assuming complete recovery. The Poisson floor of three copies per reaction therefore corresponds to 3 / 0.04 = 75 copies per mL of plasma. If extraction recovers half the input DNA, the floor rises to 150 copies per mL. An empirical whole-process LoD near that value indicates an assay working close to its theoretical limit; one several-fold higher points to losses worth investigating, perhaps in extraction or through partial inhibition. The same calculation tells the laboratory what it can gain by changing volumes: doubling the plasma input or halving the elution volume should, other things equal, roughly halve the achievable LoD. Designing the detection study The standard design for a molecular LoD study is a dilution series of a characterized target in negative native matrix, tested in many replicates across several days, with the results analyzed by probit regression. A sound plan for the BK assay might look like this. Pool plasma from donors confirmed negative for BK DNA. Spike it with material traceable to the WHO International Standard, or with a secondary standard calibrated to it, preferably using intact virus rather than plasmid so that the lysis step is exercised. Prepare five to eight concentrations bracketing the expected LoD, spaced at roughly twofold intervals, with at least two concentrations expected to yield hit rates below 95 percent and at least one expected to yield essentially 100 percent. Test each concentration in 20 or more replicates, spread over at least three days, across the operators, extraction instruments, and reagent lots that will be used clinically. Repeat the study in urine, because the matrix, the inhibitors, and the extraction behavior differ. Each replicate is scored as detected or not detected according to the assay's own interpretive rules, including any cycle-threshold cutoff beyond which amplification is called negative. That last point is easy to overlook. Many research groups set a late cutoff, say 40 cycles, beyond which curves are ignored; if the clinical procedure uses the same cutoff, the LoD study must apply it, and a change in the cutoff after validation changes the LoD. Probit analysis then fits a sigmoid relationship between the log concentration and the probability of detection. The fitted curve gives the concentration corresponding to 95 percent detection together with a confidence interval. A short sketch in Python shows the core calculation: import numpy as np, statsmodels.api as sm from scipy.stats import norm conc = np.array([25, 50, 100, 200, 400]) # copies/mL (illustrative) hits = np.array([6, 11, 17, 20, 20]); n = 20 X = sm.add_constant(np.log10(conc)) fit = sm.GLM(np.column_stack([hits, n - hits]), X, family=sm.families.Binomial(sm.families.links.Probit())).fit() b0, b1 = fit.params lod95 = 10 ** ((norm.ppf(0.95) - b0) / b1) The numbers in that snippet are illustrative, not data from any real assay. In practice the laboratory should also compute a confidence interval for the LoD, which statistical packages provide through the delta method or the fiducial approach, and should inspect the fit. Probit assumes a particular sigmoid shape; if hit rates at adjacent concentrations are erratic, the problem may lie in the dilution series rather than the assay, and a model will paper over it. The probit estimate is usually followed by a confirmation run: 20 or more replicates at the claimed LoD, often on a different day or lot, with the expectation that at least 19 of 20, or a comparable proportion, are detected. The confirmation is valuable, but it is weak evidence by itself. Nineteen detections in twenty replicates is consistent with a true detection rate well below 95 percent; the lower bound of a two-sided 95 percent confidence interval for 19 of 20 is roughly 75 percent. The probit analysis across the full dilution series, which uses all of the data, is the stronger basis for the claim. A laboratory that reports its LoD as "the lowest concentration at which 20 of 20 replicates were detected" without a dilution series may be right, but it has shown less than it thinks. Qualitative assays and the language of EP12 Many molecular tests report a qualitative result: detected or not detected, mutation present or absent. For these, the CLSI guideline EP12, whose third edition was published in 2023 under the title "Evaluation of Qualitative, Binary Output Examination Performance," supplies an allied vocabulary. Every qualitative test has an underlying continuous quantity, even if hidden, and a cutoff that converts it to a binary answer. Near the cutoff, repeated testing of the same sample gives a mix of positive and negative results. EP12 describes that transition with three concentrations: C50, where half the results are positive; C5, where 5 percent are positive; and C95, where 95 percent are positive. The C5 to C95 interval is a direct measure of the imprecision of the qualitative call. A narrow interval means the test switches cleanly from negative to positive; a wide one means that samples across a broad range of concentrations will give inconsistent answers on repeat testing. For a molecular qualitative test whose cutoff is effectively "any amplification," the C95 and the LoD are the same concept approached from different directions, and the probit study provides both. For tests with an explicit cutoff above zero, such as a variant allele fraction threshold for calling a somatic mutation, the C5 to C95 framing is more natural. Suppose an assay for a tumor mutation reports "detected" when the variant allele fraction exceeds 1 percent. Samples near that threshold will flip between results, and the laboratory should characterize the width of that zone, because a clinician receiving "not detected" on a sample with a true fraction of 1.5 percent needs to know that such results are possible. From detection to quantitation For a quantitative assay, detection is only the first threshold. The lower limit of the reportable range should be the lowest concentration at which the numerical result meets the laboratory's accuracy goal. EP17 describes several approaches; the one most suited to molecular assays estimates bias and imprecision at several low concentrations and identifies the lowest at which total error falls within the goal. Continuing the BK example, suppose the laboratory's goal is that at the LoQ the standard deviation of log10-transformed results not exceed 0.25 log10 and the mean result lie within 0.3 log10 of the assigned value. It tests samples at 150, 300, 600, and 1,200 copies per mL, each in 20 replicates over five days, and finds that the standard deviation is 0.38 log10 at 150, 0.26 at 300, 0.19 at 600, and 0.15 at 1,200, with acceptable bias at all four. The LoQ under those goals would be 600 copies per mL, even if the LoD were 150. Results between the two would be reported qualitatively, typically as "detected, below the lower limit of quantitation" or with a phrase such as "less than 600 copies/mL, detected." The choice of goal drives the choice of LoQ, which is exactly why the goal must be set before the data are seen. A laboratory that picks its precision goal after looking at the numbers can move the LoQ wherever it likes. The log transformation deserves comment. Viral loads and other nucleic acid quantities span orders of magnitude, and their variability scales roughly with concentration, so standard deviations on the log scale are more stable across the range than standard deviations in copies per mL. Reporting precision as a log10 standard deviation, or as a coefficient of variation computed from it, keeps the numbers interpretable. A log10 standard deviation of 0.15 corresponds to a coefficient of variation on the linear scale of roughly 35 percent, which sounds alarming to someone used to chemistry analytes but is typical of good quantitative PCR at moderate concentrations. Common mistakes and how to avoid them Several errors recur in detection studies transferred from research settings. The most frequent is determining the LoD on purified nucleic acid spiked into water or buffer, then reporting that figure as the clinical LoD. As shown above, the whole-process limit can be several-fold higher, and the clinical claim must rest on matrix-spiked or native specimens taken through the entire procedure. A second is using too few replicates. Five replicates at each concentration cannot distinguish a 95 percent detection rate from an 80 percent one. Twenty per level is a practical minimum, and establishing a claim that will appear on reports justifies more. A third is neglecting variants. An LoD determined on one genotype applies to that genotype. BK polyomavirus has several genotypes and subtypes, and sequence polymorphisms under a primer or probe can reduce amplification efficiency enough to raise the LoD considerably for some strains, or to cause dropout. The laboratory should confirm, at least at concentrations near the LoD, that the assay detects representative genotypes with comparable efficiency. That work overlaps with the inclusivity studies discussed in Chapter 4. A fourth is ignoring the quantitative scale. Copies per mL established with a laboratory's in-house plasmid standard are not the same unit as international units per mL defined by the WHO standard, and the conversion factor between them is assay-specific. A laboratory that reports in international units must establish its LoD in international units, using material traceable to the international standard. A fifth is failing to repeat the study when something material changes. A new extraction platform, a different elution volume, a revised cycle-threshold cutoff, or new primer lots with different purity can all move the LoD. The change control procedures in Chapter 9 should specify which changes require a new detection study, and the answer is usually more of them than a busy laboratory would like. Reporting what the limits mean Analytical sensitivity is ultimately communicated to clinicians through the report. A "not detected" result on a test with an LoD of 150 copies per mL does not mean the virus is absent; it means that, if present, it is probably below about that concentration, with some chance of a miss even above it. Reports and test directories should state the LoD and the LoQ in the units used for reporting, and interpretive comments should explain the meaning of results that are detected but below the quantitation limit. Clinicians comparing results from different laboratories or different assays need to know that the limits differ. The effort spent establishing these numbers is wasted if they are hidden in a validation binder and never reach the person making the clinical decision. Hashtags: #ClinicalDiagnosticAssayValidation #ClinicalLaboratoryValidation #CLIA #CAPAccreditation #LaboratoryDevelopedTests #LDTValidation #RegulatoryBenchmarks #ClinicalLaboratoryImprovementAmendments #AnalyticalValidation #PerformanceSpecifications #AnalyticalSensitivity #AnalyticalSpecificity #PrecisionStudies #AccuracyAssessment #ReportableRange #ReferenceIntervals #LimitOfDetection #LimitOfQuantitation #MethodComparison #LinearityStudies #InterferenceTesting #MatrixEffects #QualityControl #CLSIStandards #FutureOfClinicalDiagnostics

Download the Book (PDF): Introduction Put a tablet of paracetamol in a near-infrared spectrometer and you will get, in a few seconds, a spectrum of several hundred or several thousand absorbance values. Nowhere in that spectrum is there a peak that belongs to paracetamol alone. The drug's overtone and combination bands sit on top of those from the lactose, the microcrystalline cellulose, the starch, the magnesium stearate and whatever water the tablet has picked up since it was pressed. The bands are broad, tens of nanometres wide, and they overlap so thoroughly that the spectrum looks like a set of gentle hills rather than the sharp, assignable features a student learns to expect from an infrared spectrum of a pure liquid. Worse, the whole curve tilts and lifts from one tablet to the next because the tablets differ in thickness, surface finish and particle size, and light scatters differently in each. And yet near-infrared spectroscopy is used every day, in regulated pharmaceutical plants, to measure the active ingredient in tablets like this one, non-destructively and without a solvent in sight. The same is true of protein in wheat at grain elevators, fat in milk at dairies, octane number in refinery streams and moisture in almost everything. The instrument is not doing anything clever. The cleverness is in the arithmetic applied afterwards: a sequence of operations that removes the physical distortions from the spectra, finds the directions in which the spectra actually vary, and relates that variation to reference values measured by a slower, trusted method. That arithmetic is chemometrics. The word was coined in the early 1970s, by the Swedish chemist Svante Wold, and the field was organised soon afterwards by Wold and the American analytical chemist Bruce Kowalski, who founded the International Chemometrics Society in 1974. Its intellectual roots are older. Principal component analysis goes back to Karl Pearson in 1901 and Harold Hotelling in 1933. Partial least squares was developed by Herman Wold, Svante's father, as a method for econometric path models, and adapted by his son and by Harald Martens into the calibration tool that chemists now use. The Savitzky-Golay filter, which appears in nearly every modern spectroscopy package, was published in Analytical Chemistry in 1964 by Abraham Savitzky and Marcel Golay, two scientists at the instrument maker Perkin-Elmer, and remains one of the most cited papers that journal has ever printed. What this book argues The central claim of this book is simple to state and surprisingly easy to forget in practice. A spectrum of a real sample is a sum of chemical signal and physical nuisance, and every step of a chemometric analysis is justified only to the extent that it separates the two. Baseline correction and scatter correction exist to remove variation that has nothing to do with composition. Savitzky-Golay derivatives exist to suppress broad, slowly varying offsets while preserving the narrower features that carry chemical information. Principal component analysis exists to show you how many independent sources of variation the data really contain, and which samples do not fit the pattern. Partial least squares regression exists to find the directions in the spectra that predict the property you care about, while ignoring the directions that merely vary. The corollary is that none of these tools is chemistry-blind in the way their mathematical descriptions suggest. A model that predicts well on a held-out set is not thereby a good model; it may be exploiting a correlation between the analyte and some irrelevant physical property that happened to hold in the calibration samples and will break the day that property changes. The discipline of chemometrics, as opposed to its mathematics, lies in checking at every stage that the variation the model uses is variation the chemist can account for. The mathematics is necessary. It is not sufficient. Who this book is for This book is written for analytical chemists and for scientists in adjacent fields who work with spectra, chromatograms or other signals recorded at many channels at once. It assumes familiarity with Beer's law, with the idea of a calibration curve, and with basic statistics: means, standard deviations, least-squares straight lines. It does not assume any linear algebra beyond what can be explained in a paragraph, and where matrices appear they are introduced as what they are in the laboratory, which is tables of numbers with samples as rows and wavelengths as columns. It is not a software manual. The operations described here are available in commercial packages such as those from Camo, Eigenvector Research and Sartorius, in the chemometric toolboxes for MATLAB, in R and in Python. The book describes what those operations do, what choices they force on the analyst and what can go wrong, so that the reader can use any of them with judgement rather than by default settings. It is also deliberately narrow. The field of chemometrics includes experimental design, multivariate curve resolution, multiway methods such as PARAFAC, classification techniques, and increasingly machine-learning methods from outside the chemical tradition. Those are touched on only where they illuminate the main line, which runs from a raw spectrum to a validated quantitative prediction. How the book is organised Chapter 1 begins where every calibration begins, with the straight-line model of Beer's law, and shows exactly why it fails for overlapping signals and what the multivariate alternative has to achieve. It introduces the data matrix and the bilinear model that underlies everything that follows. Chapters 2 and 3 deal with preprocessing. Chapter 2 catalogues the physical and instrumental effects that corrupt spectra, such as baseline drift, multiplicative scatter and path-length variation, and the methods used to remove them: polynomial and asymmetric least squares baselines, standard normal variate, multiplicative scatter correction and detrending. Chapter 3 is devoted to the Savitzky-Golay filter, explaining how a moving polynomial fit produces both smoothing and derivatives, why window width and polynomial order trade noise against distortion, and how to choose them. Chapters 4 and 5 treat principal component analysis. Chapter 4 builds it from geometry: a cloud of samples in a space with one axis per wavelength, and the rotated axes that describe that cloud most economically. Chapter 5 turns to its use as a diagnostic tool, covering scores, loadings, residuals, Hotelling's T² and Q statistics, and the detection of outliers and structure that nobody expected. Chapters 6 and 7 are about regression. Chapter 6 explains why ordinary multiple regression cannot be applied directly to spectra, how principal component regression addresses the problem, and how partial least squares improves on it by building components that are relevant to the property being predicted. Chapter 7 is about making a calibration that works outside the laboratory in which it was built: choosing samples, choosing the number of latent variables, validating honestly, and reporting figures of merit that mean something. Chapter 8 addresses interpretation and maintenance: what the regression vector and variable importance measures do and do not tell you, how variable selection helps and harms, why models degrade when instruments or raw materials change, and how calibrations are transferred and updated. It closes with a short treatment of PLS used for classification, a technique so widely misused that it deserves specific warnings. The conclusion draws the threads together into a set of working principles for anyone who builds or relies on a multivariate calibration. A brief note on notation. Matrices are written as bold capital letters, X for the spectra and Y or y for the reference values. A sample is a row, a variable is a column. Where a symbol is needed for the number of samples it is n, for the number of variables (wavelengths) it is p, and for the number of components or latent variables it is A. The transpose of a matrix is marked with a superscript T, as in Xᵀ. That is all the notation the book requires. Chapter 1: Beyond Beer's Law Every analytical chemist learns calibration the same way. You make up a series of standards of known concentration, measure the absorbance of each at the wavelength of maximum absorption, plot absorbance against concentration, fit a straight line by least squares, and read unknowns off the line. The procedure rests on the Beer-Lambert law, which states that absorbance A at a given wavelength equals the product of the molar absorptivity ε of the absorbing species, the path length b and the concentration c. It is a beautiful law, and when its assumptions hold it is extraordinarily reliable. This chapter is about what happens when they do not, and about the shape of the solution. The hidden assumption of univariate calibration A single-wavelength calibration assumes that the signal at the chosen wavelength comes from the analyte and nothing else, or that whatever else contributes is constant across standards and unknowns so that it can be subtracted as a blank. In the teaching laboratory this is arranged deliberately. The iron is complexed with 1,10-phenanthroline so that it absorbs at 510 nm where the other constituents of the solution do not; the reagent blank takes care of any residual background. In a real sample the assumption fails in one of three ways. The first is spectral overlap: another constituent absorbs at the chosen wavelength, and its concentration varies from sample to sample. The second is background variation: the baseline under the analyte's band moves because of turbidity, scattering, reflection losses or instrumental drift. The third is matrix effect: the analyte's own absorptivity changes with its environment, for instance through hydrogen bonding in near-infrared spectra, so that the same concentration gives a different signal in a different matrix. The univariate calibration has no way to tell any of these apart from a change in analyte concentration. It reads every change in absorbance at its wavelength as analyte, and so every interference becomes an error. The traditional remedies are chemical and chromatographic. You separate the analyte from the interferences, or you derivatise it into something that absorbs where nothing else does, or you choose a more selective technique. These remedies work, and for trace analysis they are often still the right answer. But they are slow, they consume reagents and solvents, and they destroy the sample. The alternative that chemometrics offers is to accept the overlap and resolve it mathematically, by measuring at many wavelengths at once. Why more wavelengths help Consider the simplest possible mixture: two absorbing compounds, P and Q, in a non-scattering solvent. At any wavelength the absorbance of the mixture is the sum of the two contributions, because Beer's law is additive for independent absorbers. At wavelength 1 the absorbance is ε(P,1)·b·c(P) + ε(Q,1)·b·c(Q); at wavelength 2 it is ε(P,2)·b·c(P) + ε(Q,2)·b·c(Q). If the four absorptivities are known from pure standards, the two measured absorbances give two linear equations in two unknowns, and the concentrations can be solved exactly, provided the two compounds' spectra are not simply scaled copies of each other. This is Vierordt's method, published in the nineteenth century and still taught for two-component pharmaceutical assays. The generalisation is immediate. With k absorbing components and measurements at p wavelengths, where p is at least k, the mixture spectrum is a weighted sum of the k pure-component spectra, the weights being the concentrations. Writing the pure spectra as the rows of a matrix K and the concentrations as a vector c, the mixture spectrum is x = cᵀK, apart from noise. When p exceeds k the system is overdetermined, and the concentrations are estimated by least squares. This approach, known as classical least squares or the K-matrix method, is exactly what Beer's law implies for a multi-component system, and in favourable cases it works very well. Its great advantage over a single wavelength is that the redundancy averages down noise, and the use of the whole spectral shape distinguishes compounds that coincide at any one wavelength. Classical least squares has one crippling requirement, however. Every source of spectral variation must be represented in K. If there is a third compound in the unknown that was not in the calibration, or a baseline shift, or a temperature-dependent band shift, the least-squares fit will distribute that unexplained signal among the components it does know about, and the concentrations will be wrong. For a synthetic mixture of known dyes this is manageable. For a tablet, a grain kernel or a sample of blood serum it is hopeless: nobody can write down the spectrum of every constituent, let alone the spectral effects of particle size and packing density. Turning the problem around The decisive step in multivariate calibration is to stop trying to model the spectrum from the concentrations and instead model the concentration from the spectrum. This is called inverse calibration. You write the concentration of the analyte as a weighted sum of absorbances at many wavelengths, y = b(1)·x(1) + b(2)·x(2) + ... + b(p)·x(p) + b(0), and estimate the weights, the regression coefficients, from a set of calibration samples for which both the spectra and the reference concentrations are known. The advantage of the inverse model is that it needs to know only one thing about each calibration sample: the concentration of the analyte of interest. It does not need the concentrations of the interferents, their spectra, or any physical description of the baseline. It needs only that the interferents and baseline effects vary in the calibration set in the same way they will vary in future samples. If they do, the regression can learn weights that respond to the analyte and cancel the rest. A negative weight at a wavelength where an interferent absorbs, balancing a positive weight elsewhere where both absorb, is the mathematical form of this cancellation. Stated plainly, the inverse model does not remove the need for chemical knowledge. It moves it. Instead of knowing the spectrum of every constituent, the analyst must know which constituents and physical effects vary in practice, and make sure the calibration samples span that variation. This is why sample selection, discussed in Chapter 7, matters more in multivariate calibration than in any univariate method. A calibration set that happens to have the analyte correlated with an interferent will produce a model that measures the interferent and calls it the analyte, and nothing in the regression statistics will warn you. The idea that the model only needs the interferents to vary has a precise form in the concept of net analyte signal, introduced by Avraham Lorber in 1986. The net analyte signal is the part of the analyte's spectrum that is orthogonal to, that is, cannot be reproduced by any combination of, the spectra of everything else that varies. Only that part carries information about the analyte in a mixture. If the analyte's spectrum is very similar to an interferent's, the net analyte signal is small, the model has to amplify it, and noise is amplified with it. Selectivity in multivariate calibration is therefore not a yes-or-no property of a wavelength but a continuous property of the whole spectrum, measured by how much of the analyte signal survives once the interferences are projected away. A constructed example A small invented example makes the difference between the classical and inverse approaches concrete. Suppose an analyst wants to determine a dye, call it P, in aqueous samples that also contain a second dye, Q, whose band overlaps P's heavily. At the wavelength of P's maximum, Q contributes about a third as much absorbance per unit concentration as P does. The concentration of Q in real samples varies between zero and a level comparable with P. A univariate calibration at P's maximum, built from standards of pure P, will read every sample high by an amount proportional to its Q content. If Q is typically present at about the same concentration as P, the bias will be of the order of thirty per cent, and it will vary from sample to sample, so it cannot be corrected by a fixed blank. The calibration line itself will look perfect, with a correlation coefficient of 0.9999, because the standards contained no Q. This is the most dangerous feature of univariate calibration against clean standards: its quality statistics describe the standards, not the samples. A classical least squares approach would record the full spectra of pure P and pure Q, and fit each unknown as a combination of the two. This works, provided nothing else in the samples absorbs. If a third substance appears, such as a small amount of a yellow degradation product, the fit will attribute its absorbance partly to P and partly to Q in whatever proportions best mimic its shape, and both estimates will be biased. An inverse approach would take, say, thirty real samples, measure P in each by a reference method such as HPLC, record their full spectra, and regress P on the spectra. The resulting regression vector will have large positive weights where P absorbs strongly relative to Q, and negative weights where Q absorbs strongly relative to P, arranged so that Q's contribution cancels. If some of the thirty samples contained the degradation product, at levels uncorrelated with P, the regression will have learned to cancel that too, without anyone having had to identify it. If none did, the model will be as vulnerable to it as the classical approach was. The inverse model is not magic; it is a way of learning cancellations from examples, and it can learn only from the examples it is given. Two further points emerge from the example. First, the inverse model's accuracy can never be better than that of the reference method used to measure P in the calibration samples, since those values are the targets it is trained on. An HPLC assay with a relative standard deviation of one per cent sets a floor under the spectroscopic prediction error. Second, the example has a natural dimensionality of two, or three with the degradation product. However many wavelengths the spectrometer records, the useful information lies in a two- or three-dimensional subspace. Finding that subspace is the business of the chapters on PCA and PLS. The data matrix The raw material of every method in this book is the data matrix X. Each row is one sample and each column is one variable: an absorbance at one wavelength, an intensity at one Raman shift, a signal at one retention time, one chemical shift in an NMR spectrum. For a typical near-infrared calibration, X might have 100 to 300 rows and between 500 and 2,000 columns. For Raman or mid-infrared spectra the number of columns is similar; for hyperspectral images, where every pixel is a sample, the number of rows can run into millions. Alongside X sits the response, y, a column of reference values measured by a conventional method: HPLC assay, Kjeldahl nitrogen, Karl Fischer water, gravimetric fat. When several properties are predicted at once the responses form a matrix Y. Two features of spectral data matrices shape everything that follows. The first is that p is usually larger than n, often much larger: there are more wavelengths than samples. The second is that the columns are highly correlated. Neighbouring wavelengths in a smooth spectrum carry nearly the same information; a broad band spreads the signal of one constituent across dozens or hundreds of columns. Both features make ordinary least squares regression impossible, for reasons Chapter 6 explains, and both point towards the same solution: summarise the many correlated columns by a few new variables that capture how the spectra actually vary. The bilinear model The observation that a mixture spectrum is a sum of component spectra, weighted by concentrations, has a direct consequence for the whole data matrix. If there are k independent sources of variation in the samples, and each contributes a fixed spectral profile scaled by some sample-specific amount, then X can be written as the product of two thin matrices, one n rows by k columns holding the amounts and one k rows by p columns holding the profiles, plus a residual matrix of noise. In the notation of Beer's law this is X = CK + E. In the notation of principal component analysis, which Chapter 4 develops, it is X = TPᵀ + E, where T holds the scores and P the loadings. This is called a bilinear model because it is linear in the amounts for fixed profiles, and linear in the profiles for fixed amounts. It is the single most important idea in chemometrics. It says that a data matrix with a thousand columns may have only five or ten independent directions of meaningful variation, and that everything else is noise. It says that the job of analysis is to find those few directions and work with them. The sources of variation need not be chemical constituents. A baseline offset that varies from sample to sample is a source: its profile is a flat line, and its amount is the size of the offset. A linear baseline tilt is another. Multiplicative scatter, which scales the whole spectrum up or down, behaves like a source whose profile is the mean spectrum. Temperature, which shifts and broadens hydrogen-bonded bands, adds one or more further profiles. In a real data set the k directions in the bilinear model are a mixture of chemical and physical effects, and the rotation that PCA or PLS finds will generally not align any one component with any one physical cause. That is a limitation, and it is also why preprocessing matters: every physical effect that can be removed before modelling is one fewer direction the model has to spend on nuisance. What has to go right The rest of this book is a series of answers to the question of how to get from a raw X and y to a model that predicts correctly for samples it has never seen. It is worth listing what has to go right, because each item corresponds to a stage of the analysis and to a characteristic way of failing. · The spectra must contain information about the analyte: a net analyte signal large enough, compared with noise, to support the required precision. No amount of mathematics can extract a component that does not absorb, or whose signal is swamped by others. · The physical and instrumental distortions of the spectra must be removed, or at least made smaller than the chemical variation of interest. This is the job of preprocessing, the subject of Chapters 2 and 3. · The dimensionality of the data must be reduced to the few directions that carry real variation, without discarding directions that carry analyte information and without keeping directions that carry only noise. This is the job of PCA and PLS, the subjects of Chapters 4 to 6. · The calibration samples must span the variation that future samples will show, in analyte, in interferents and in physical properties, and the reference values must be accurate. This is a matter of experimental design and sampling, treated in Chapter 7. · The model's predictive performance must be estimated on samples that played no part in building it, and the estimate must reflect how the model will be used. This is validation, also treated in Chapter 7. · The model must be monitored and maintained, because instruments age, raw materials change and processes drift. This is the subject of Chapter 8. It is tempting to treat these as a checklist to be worked through once. In practice they interact. A preprocessing choice changes how many components the model needs; a new outlier found in the PCA sends you back to the laboratory to find out why; a validation result that looks too good sends you back to ask whether the calibration and validation samples were really independent. Good chemometric practice is iterative, and each chapter that follows should be read with the others in mind. Chapter 2: Baselines, Scatter and the Physics of Nuisance Before any multivariate model is built, the spectra are almost always transformed. Analysts call this preprocessing, and the choices it involves are among the most consequential in the whole analysis. A well-chosen preprocessing step can halve the number of components a model needs and substantially reduce its prediction error. A badly chosen one can remove the very signal the model was meant to find, or introduce artefacts that the model then learns. The difficulty is that the same algebraic operation can be helpful for one data set and harmful for another, so preprocessing cannot be chosen by rule. It has to be chosen by understanding what physical effects are present in the spectra and what each method does to them. This chapter is about the effects that act on the spectrum as a whole or over broad regions: baselines and scatter. Chapter 3 deals with noise and with the Savitzky-Golay filter, which addresses both noise and baselines through smoothing and differentiation. Where baselines come from A baseline, in the sense used here, is any contribution to the measured signal that varies smoothly with wavelength and is unrelated to the absorption bands of interest. The causes differ between techniques. In transmission spectroscopy of liquids, baselines arise from differences in cell windows, from fouling, from bubbles and from small changes in the reference measurement. They tend to be offsets or gentle slopes. In diffuse reflectance near-infrared spectroscopy of solids, which is the workhorse of industrial chemometrics, the dominant effect is light scattering. Light entering a powder or a tablet is scattered many times before it re-emerges, and the effective path length through the sample depends on particle size, packing density and surface texture. A coarse powder scatters less and lets light travel further, so it appears to absorb more at every wavelength. The result is a set of spectra that differ from each other by offsets, slopes and multiplicative factors that have nothing to do with composition. In Raman spectroscopy the largest baseline is fluorescence, which can be orders of magnitude stronger than the Raman signal and appears as a broad, curved background whose shape varies with the sample and even with how long the sample has been illuminated, since fluorescent impurities photobleach. In mass spectrometry and chromatography, baselines come from column bleed, from gradient changes in the mobile phase and from detector drift. It is useful to classify baseline effects by their mathematical form. An additive offset raises or lowers the entire spectrum by a constant. An additive slope adds a straight line, and more generally a curved baseline adds a smooth function such as a low-order polynomial. A multiplicative effect scales the entire spectrum by a sample-specific factor, as a change in effective path length does. Real spectra usually show a combination: in diffuse reflectance, a measured spectrum is often well described as a + b·m + (small polynomial), where m is some reference spectrum, a is an offset and b a multiplicative scale factor, both varying from sample to sample. Why nuisance matters to a multivariate model It might seem that a multivariate model could simply learn to ignore baselines. Chapter 1 argued that the inverse model cancels any interference that varies in the calibration set, and a baseline is just another interference. This is true in principle and costly in practice, for three reasons. First, every independent physical effect uses up a component. If offset, slope and multiplicative scatter all vary, a PLS model will typically need three or more latent variables to account for them before it gets to the chemistry. Each extra component adds estimation uncertainty and increases the risk of overfitting. Second, multiplicative effects are not additive, and the bilinear model handles them badly. If the spectrum of every sample is scaled by a factor b, then the absorbance contributed by the analyte is scaled too. The model cannot correct this by adding a component; a multiplicative distortion of a linear signal is a nonlinearity, and linear models approximate nonlinearities only by spending further components. Third, physical effects are often large compared with the chemical signals of interest. In near-infrared spectra of powders, scatter frequently accounts for more than ninety per cent of the total variance between samples. A PCA of such spectra is dominated by scatter in its first component, and the chemical information is relegated to later components with small variance, where it is harder to see and easier to confuse with noise. For all these reasons it is usually worth removing physical effects explicitly when their form is known. The methods below are the standard ones. Offset and polynomial baseline correction The simplest correction subtracts a constant from each spectrum, usually the value at a wavelength or the mean over a region where no bands are expected. A slightly more elaborate version fits a straight line or low-order polynomial through a few chosen baseline points and subtracts it. These methods are transparent and easy to justify when the analyst knows where the baseline lies, as in mid-infrared spectra of liquids with clear band-free regions. Their weakness is that choosing baseline points requires judgement for each data set, and in near-infrared spectra there are no truly band-free regions. A related method, detrending, proposed by Barnes, Dhanoa and Lister in 1989 alongside standard normal variate, fits a second-degree polynomial to the whole spectrum as a function of wavelength and subtracts it. Because the fit uses all the data, including the bands, it removes some chemical signal along with the baseline, but for broad NIR spectra where bands are small wiggles on large curvature the loss is usually tolerable. Asymmetric least squares and its relatives For spectra with sharp bands on a curved background, such as Raman spectra with fluorescence, the most widely used baseline methods today are based on penalised smoothing with asymmetric weights. The idea, developed by Paul Eilers and Hans Boelens at Leiden in the mid-2000s and built on Eilers's Whittaker smoother described in Analytical Chemistry in 2003, is elegant. A smooth curve z is fitted to the spectrum x by minimising the sum of two terms: a weighted sum of squared differences between x and z, and a penalty λ times the sum of squared second differences of z. The penalty keeps z smooth; larger λ gives a stiffer curve. The weights are what make the method asymmetric. Points where the spectrum lies above the fitted curve, which are likely to be peaks, get a small weight p, typically between 0.001 and 0.05. Points below the curve get weight 1 − p. The fit is repeated, updating the weights each time, until it converges. Because peaks are almost ignored and the background is followed closely, the curve settles along the lower envelope of the spectrum, and subtracting it leaves the bands on a flat baseline. The method has two parameters, the smoothness λ and the asymmetry p, and both matter. Too small a λ lets the baseline follow broad bands and eat into them; too large a λ leaves curvature behind. The usual approach is to try values spanning several orders of magnitude, typically 10² to 10⁹, and to inspect the corrected spectra for a few representative samples. Variants such as adaptive iteratively reweighted penalised least squares and asymmetrically reweighted penalised least squares change how the weights are updated to reduce sensitivity to the choice of p and to handle noisy baselines more gracefully. None removes the need to look at the result. Asymmetric least squares is a baseline method, not a scatter correction. It subtracts an additive background and does nothing to multiplicative effects. For Raman data it is often followed by a normalisation step for that reason. Standard normal variate Standard normal variate, abbreviated SNV, was introduced for near-infrared diffuse reflectance by Barnes, Dhanoa and Lister in Applied Spectroscopy in 1989. It could hardly be simpler. Each spectrum is centred by subtracting its own mean across all wavelengths, and then scaled by dividing by its own standard deviation across all wavelengths. After SNV every spectrum has mean zero and standard deviation one. The subtraction removes additive offsets. The division removes multiplicative scale. Together they correct the simplest form of scatter, in which each spectrum is a + b·s, where s is the underlying scatter-free spectrum. SNV operates on each spectrum independently, which is a practical advantage: a new sample can be corrected without reference to the calibration set, and adding or removing samples from the calibration set does not change the preprocessing of any other sample. SNV has two weaknesses that follow directly from its definition. Because the mean and standard deviation are computed over the whole spectrum, they include the chemical bands. If a sample contains more of a strongly absorbing constituent, its standard deviation increases, and SNV scales that sample down, partly undoing the chemical effect. In most NIR applications this distortion is small because the chemical variation is small compared with the scatter, but it is real and it can introduce nonlinearity. Second, SNV's result depends on the wavelength range used. Cutting off a noisy region at one end of the spectrum changes every sample's mean and standard deviation and therefore changes the corrected spectra everywhere. The range should be fixed before SNV is applied, and never changed afterwards without recomputing. Multiplicative scatter correction Multiplicative scatter correction, MSC, was introduced by Paul Geladi, Douglas MacDougall and Harald Martens in Applied Spectroscopy in 1985 for near-infrared reflectance spectra of meat. It addresses the same model as SNV, a + b·s, but estimates a and b differently. A reference spectrum m is chosen, usually the mean spectrum of the calibration set. Each sample spectrum x is regressed on m by ordinary least squares across wavelengths, giving an intercept a and slope b such that x ≈ a + b·m. The corrected spectrum is (x − a)/b. The logic is that most of the difference between a sample spectrum and the mean is scatter, so when the sample is plotted against the mean, wavelength by wavelength, the points fall close to a straight line whose intercept and slope measure the offset and multiplicative scatter. The chemical differences appear as small deviations from the line, and they survive the correction. MSC and SNV are closely related. It can be shown that, when the reference spectrum is the mean and the scatter model holds, the two give results that differ approximately by a linear transformation, and models built on either usually perform similarly. The differences are practical. MSC depends on the reference spectrum, so it ties new samples to the calibration set: a new sample is corrected against the old mean. That is appropriate for a stable application and awkward when the calibration set is updated. MSC also shares SNV's contamination by chemical signal, since the regression uses all wavelengths; if the analyte's band is strong, samples rich in analyte will get a larger b and be scaled down. Extended multiplicative signal correction The contamination problem led Harald Martens and colleagues to develop extended multiplicative signal correction, EMSC, in the early 1990s and in a series of later papers. EMSC fits each spectrum not just as a + b·m but as a + b·m plus additional terms: polynomial functions of wavelength to model curved baselines, and, crucially, known spectra of interferents or even of the analyte. By including the analyte's pure spectrum as a regressor, the method estimates the scatter parameters while allowing the analyte signal to vary, and so avoids scaling it away. EMSC is the most principled of the scatter corrections because it makes the physical model explicit. Its cost is that the analyst must decide which terms to include, and must have spectra for any chemical components added to the model. In biological infrared spectroscopy, where Mie scattering from cells produces strongly curved, wavelength-dependent distortions, extended versions of EMSC incorporating physical scattering models have become standard. In routine NIR calibration, plain MSC or SNV followed by a derivative remains more common. Normalisation Normalisation divides each spectrum by a single number: its total area, its maximum, its Euclidean length, or the intensity of a chosen internal-standard band. It corrects multiplicative effects but not additive ones. Normalisation is routine in Raman spectroscopy, where laser power, focus and sample positioning change the overall intensity, and in mass spectrometry and NMR-based metabolomics, where sample dilution varies. In metabolomics, probabilistic quotient normalisation, described by Dieterle and colleagues in Analytical Chemistry in 2006, has become popular because it estimates the dilution factor from the median of the ratios between a sample and a reference, which is robust to changes in a few metabolites. Normalisation to total signal carries a particular danger that is easy to overlook: closure. After dividing by the total, the variables of each sample sum to a constant, so an increase in one must be matched by decreases in the others. If one constituent rises sharply, every other constituent appears to fall, even though nothing else has changed. For spectroscopic calibration the effect is usually minor, but in compositional data such as metabolite profiles it can generate spurious negative correlations. Comparing the methods The corrections described above differ in which physical effects they remove, whether they process each spectrum independently, and how they interact with the chemical signal. Table 1 sets these differences side by side for the methods most often used in practice. Table 1. Common baseline and scatter corrections compared. Method Removes Per-spectrum? Main risk Offset or polynomial subtraction Additive offset, slope or curve Yes Removing band intensity if baseline points are misplaced Detrending Quadratic curvature across the range Yes Distorting broad bands Asymmetric least squares Smooth additive background Yes Baseline following broad bands if λ is too small Standard normal variate Offset and multiplicative scale Yes Scaling away strong chemical signal; range-dependence Multiplicative scatter correction Offset and multiplicative scale relative to a reference No, needs reference spectrum Same as SNV; tied to calibration mean Extended MSC Offset, scale, polynomial baseline, known interferents No, needs reference and model terms Wrong choice of model terms Normalisation to area or length Multiplicative scale only Yes Closure; additive effects left untouched Choosing and checking a preprocessing strategy The review of NIR preprocessing by Åsmund Rinnan, Frans van den Berg and Søren Balling Engelsen in Trends in Analytical Chemistry in 2009 is still the best single survey of these methods, and its practical advice is worth repeating. Understand the physics of your measurement before choosing a method. Look at the raw spectra, all of them overlaid, and ask what kind of variation you see: vertical offsets, fans that widen with absorbance, tilts, curvature. Choose the correction that matches the form of the nuisance, not the one used in the last paper you read. Then check the result. Overlay the corrected spectra and confirm that the physical spread has shrunk while the band shapes remain. Run a PCA before and after and look at how much variance moves out of the first component. Build models with and without the correction and compare their cross-validated errors, remembering that small differences in cross-validated error between preprocessing options are often within the noise of the validation itself. Two further points are easily forgotten. The first is order. Scatter corrections and derivatives do not commute: SNV followed by a first derivative gives a different result from a first derivative followed by SNV. The common practice in NIR is to apply the scatter correction first and the derivative second, or to use a derivative alone, but there is no universal answer, and whatever order is chosen must be applied identically to every future sample. The second is leakage. Any preprocessing step that uses information from more than one sample, such as MSC with a mean reference or column centring, must be recomputed within each cross-validation fold, using only the calibration samples of that fold. If the mean spectrum is computed once from all samples, including those that will later be used for validation, a little information about the validation samples leaks into the model and the validation error is slightly optimistic. With SNV and derivatives, which act on each spectrum alone, the problem does not arise; this is a quiet argument in their favour. The best preprocessing is the one that removes the nuisance you can identify, and nothing else. When in doubt, a method that does less, applied to spectra you have looked at carefully, is preferable to a combination of methods chosen by an automated search over every possibility. The automated search will always find a combination that minimises cross-validated error on the data at hand, and the minimum will often reflect chance rather than physics. Chapter 3: Savitzky-Golay Smoothing and Derivatives In July 1964 Analytical Chemistry published a paper by Abraham Savitzky and Marcel J. E. Golay titled "Smoothing and Differentiation of Data by Simplified Least Squares Procedures". Its idea was modest and practical. Spectra were by then being digitised, recorded as a series of equally spaced numbers rather than as a pen trace on chart paper, and chemists wanted to smooth the noise out of them and to compute their derivatives. Fitting a polynomial by least squares to every little stretch of the spectrum was, on the computers of the day, prohibitively slow. Savitzky and Golay showed that it was unnecessary: for equally spaced data, the least-squares polynomial fit at the centre of a window reduces to a fixed weighted sum of the points in the window, and the weights can be tabulated once and for all. The paper printed the tables. It became one of the most cited papers in the history of analytical chemistry, and the filter it describes sits inside almost every piece of spectroscopic software written since. The original tables contained a number of numerical errors, which were identified and corrected by Jean Steinier, Yves Termonia and Jules Deltour in the same journal in 1972. The method itself was sound, and it remains the standard way to smooth and differentiate spectra before multivariate modelling. The moving polynomial The idea is best understood by imagining it done the slow way. Take a window of, say, seven consecutive points in the spectrum, centred on point i. Fit a polynomial of chosen degree, say a quadratic, to those seven points by ordinary least squares, treating the position in the window, from −3 to +3, as the independent variable. Evaluate the fitted polynomial at the centre, position 0. That value replaces the original value of point i. Now move the window one point to the right, centre it on point i + 1, fit a new quadratic, evaluate it at the centre, and so on along the whole spectrum. This is a local polynomial regression, and it behaves in a way that a simple moving average does not. A moving average replaces each point by the mean of its neighbours, which is equivalent to fitting a polynomial of degree zero, a horizontal line, in the window. On a sloping stretch of spectrum that is fine, because the average of points on a straight line equals the value at the centre. But on a peak, where the spectrum is curved, the average of the points around the maximum is lower than the maximum itself, and the moving average flattens and broadens the peak. A quadratic or higher polynomial can follow curvature, so the Savitzky-Golay smoother preserves peak heights and widths much better for the same degree of noise reduction. The key insight of the 1964 paper is that because the x-values in every window are the same, the integers from −m to +m, the least-squares solution can be written as a fixed linear combination of the y-values. The value of the fitted polynomial at the centre is Σ c(j)·y(i + j), summed over j from −m to +m, where the coefficients c(j) depend only on the window width and the polynomial degree. The smoothing is a convolution, and it runs as fast as a moving average. Derivatives from the same fit A polynomial fitted to the window has not only a value at the centre but also a slope, a curvature and higher derivatives. The first derivative of the fitted quadratic at the centre is its linear coefficient; the second derivative is twice its quadratic coefficient. These too are fixed linear combinations of the y-values, and Savitzky and Golay tabulated them alongside the smoothing coefficients. A Savitzky-Golay derivative is therefore a smoothed derivative: it estimates the local slope or curvature from a least-squares fit over several points, which is far less noisy than the naive difference between neighbouring points. The coefficients for the derivative of order d must be divided by the sampling interval raised to the power d to give derivatives in physical units, such as absorbance per nanometre. For multivariate modelling this scaling rarely matters, since it multiplies every variable by the same constant, but it matters when derivative spectra from instruments with different sampling intervals are compared. Table 2 gives some of the most commonly used coefficient sets. They illustrate the structure of the filter: the smoothing coefficients are symmetric and include negative values at the edges, which is what allows the quadratic fit to follow curvature; the first-derivative coefficients are antisymmetric, and for a quadratic or linear fit they are simply proportional to the position in the window; the second-derivative coefficients are symmetric and sum to zero. Table 2. Savitzky-Golay convolution coefficients for quadratic (or cubic) fits, from least-squares theory. Operation Window Coefficients, from −m to +m Normaliser Smoothing 5 points −3, 12, 17, 12, −3 35 Smoothing 7 points −2, 3, 6, 7, 6, 3, −2 21 Smoothing 9 points −21, 14, 39, 54, 59, 54, 39, 14, −21 231 First derivative 5 points −2, −1, 0, 1, 2 10 First derivative 7 points −3, −2, −1, 0, 1, 2, 3 28 Second derivative 5 points 2, −1, −2, −1, 2 7 Second derivative 7 points 5, 0, −3, −4, −3, 0, 5 42 Two facts about Table 2 are worth drawing out. First, the smoothing coefficients for a quadratic fit are identical to those for a cubic fit. This is because the window is symmetric, and the cubic term, being an odd function, contributes nothing to the value at the centre. The same holds for derivatives in pairs: a first derivative from a quadratic fit equals that from a linear fit, and a second derivative from a quadratic fit equals that from a cubic. Software that offers polynomial orders two and three for smoothing is offering the same filter twice. Second, for a smoothing filter the sum of the squares of the normalised coefficients equals the centre coefficient, so for white noise the variance is reduced by that factor. The five-point quadratic smoother reduces noise variance to 17/35 of its original value, a reduction in noise standard deviation of about thirty per cent. A five-point moving average would reduce the variance to one fifth, which is much more noise reduction, but at the cost of flattening peaks. The nine-point quadratic smoother reduces variance to 59/231, roughly a quarter, and noise standard deviation to about half. What a derivative does to a spectrum Derivatives are used in chemometrics for two quite different purposes, and it helps to keep them separate. The first purpose is baseline removal. The first derivative of a constant is zero, so a first derivative removes additive offsets entirely. The second derivative of a straight line is zero, so a second derivative removes both offsets and linear slopes. A curved baseline that varies slowly compared with the bands is not removed but is greatly reduced, because the derivative of a slowly varying function is small. In near-infrared work, where offsets and slopes from scatter dominate the raw spectra, a first or second derivative often achieves most of what SNV or MSC would, and sometimes more. Derivatives do not correct multiplicative effects, which is why they are frequently combined with SNV or MSC. The second purpose is resolution enhancement. The second derivative of a band shaped like a Gaussian or Lorentzian is a narrower, inverted band with a sharp negative minimum at the original maximum and small positive side lobes on either side. Two overlapping bands that appear as a single broad hump with a shoulder in the original spectrum often appear in the second derivative as two distinct minima. Spectroscopists have used this to locate hidden bands for decades. In chemometric modelling the resolution enhancement matters because it makes the spectral profiles of different constituents less similar, which increases their net analyte signals and makes the regression's job easier. The first derivative has its own characteristic shape. A single band becomes a pair of lobes, positive on the rising side and negative on the falling side, with a zero crossing at the original maximum. This makes first-derivative spectra harder to interpret by eye, since the position of a feature is marked by a zero crossing rather than a peak. For modelling, the loss of visual clarity is unimportant. The price of differentiation Derivatives amplify noise. This follows from their nature: noise varies rapidly from point to point, and differentiation emphasises rapid variation. In the language of signal processing, a derivative filter has a gain that increases with frequency, and white noise has equal power at all frequencies, so the derivative multiplies the high-frequency noise more than the low-frequency signal. A second derivative is worse than a first. The Savitzky-Golay derivative mitigates this by combining differentiation with smoothing. The wider the window, the more points contribute to each estimate, and the less noise is amplified. For the first-derivative coefficients in Table 2, the sum of squared coefficients divided by the square of the normaliser gives the variance amplification for white noise: 10/100, or 0.1, for five points, and 28/784, about 0.036, for seven points. The wider window gives much less noise in the derivative estimate, in exchange for less ability to follow sharp features. That is the central trade-off of the filter. A wide window and a low polynomial degree give strong noise reduction but distort narrow bands, reducing their height and broadening them, and in the case of derivatives they can merge features that should be separate. A narrow window or a high polynomial degree follows sharp features faithfully but passes more noise. There is no setting that is best for every purpose, because the answer depends on the ratio between the width of the bands and the spacing of the data points. Choosing window and degree A practical rule, which goes back to the early work on the filter and appears in most reviews, is that the window should be comparable to or smaller than the full width at half maximum of the narrowest band that needs to be preserved. If the bands of interest are about 20 nm wide and the spectrum is sampled every 2 nm, a band spans about ten points, and a window of seven to eleven points is a sensible starting place for a quadratic fit. A window much wider than the band will distort it noticeably; a window much narrower will provide little noise reduction. Near-infrared spectra, with bands often 30 to 100 nm wide, tolerate wide windows. Windows of 11 to 25 points are common, and the noise reduction they provide is valuable because NIR derivatives would otherwise be very noisy. Raman and mid-infrared spectra, with bands of a few wavenumbers to a few tens of wavenumbers wide, need narrower windows relative to the sampling interval. Chromatographic and mass spectrometric data, with very sharp peaks, need the narrowest windows of all, and sometimes are better served by other smoothing methods. Raising the polynomial degree allows the filter to follow sharper features in a given window, so that a quartic fit in a wide window can preserve peaks nearly as well as a quadratic fit in a narrow one, with more noise reduction. But higher degrees also pass more noise, so the gain is partial. In chemometric preprocessing, quadratic fits are by far the most common, and a sensible procedure is to fix the degree at two and vary the window. The choice can be checked in two ways. The first is to look: plot the smoothed or differentiated spectra for a few samples, compare them with the raw spectra, and see whether real features have been lost or noise remains. Subtracting the smoothed spectrum from the raw one should leave something that looks like featureless noise; if the residual shows structure at the positions of bands, the window is too wide. The second is to build calibration models with several window widths and compare cross-validated errors. This second approach is legitimate and common, but it should be applied to a small grid of physically sensible choices rather than a fine search, for the reasons given at the end of Chapter 2. Edges and gaps A symmetric window cannot be centred on the first m or last m points of a spectrum, since there are not enough neighbours. Software handles this in several ways: by dropping the end points, by padding the spectrum with repeated or mirrored values, or by fitting the polynomial to the first or last full window and evaluating it at the off-centre positions. The last approach, for which Peter Gorry published general coefficient formulas in Analytical Chemistry in 1990, is the most defensible, but the end points of a derivative spectrum are always noisier than the interior. The simplest safe practice is to record a slightly wider range than needed and to discard the ends after filtering. A variant that is still common in near-infrared work is the gap-segment derivative associated with Karl Norris, a pioneer of NIR analysis at the United States Department of Agriculture. Rather than fitting a polynomial, it averages the data over a segment of points on either side of a gap and takes the difference between the two averages as the derivative estimate. Norris derivatives and Savitzky-Golay derivatives are both linear filters and behave similarly when their parameters are matched; they differ in the exact shape of their frequency response. Instrument software from the older NIR tradition often offers the Norris form, and published NIR methods frequently specify parameters such as "2,10,10,1", meaning derivative order, gap, segment and a secondary smoothing, a notation that is meaningless outside that software family and should be translated into explicit filter parameters when a method is transferred. Derivatives in the full preprocessing chain In a typical near-infrared calibration, the preprocessing chain might consist of SNV to remove multiplicative scatter, a Savitzky-Golay first or second derivative with a window of 11 to 21 points and a quadratic fit to remove residual baselines and sharpen bands, and then mean-centring of each variable across samples before PCA or PLS. For Raman spectra, a common chain is cosmic-ray spike removal, asymmetric least squares baseline correction, light Savitzky-Golay smoothing, and normalisation to area or to an internal standard band. For mid-infrared spectra of liquids, derivatives may not be needed at all, and a simple offset correction and mean-centring can be enough. Mean-centring deserves a word of its own, because it is applied so routinely that it is easy to forget it is a choice. Subtracting the mean spectrum of the calibration set from every spectrum moves the origin of the data to the centre of the sample cloud. Without it, the first component of a PCA or PLS model describes mostly the mean spectrum itself, which is the same for every sample and so carries no information about differences between them. Mean-centring is almost always appropriate for spectral data. Scaling each variable to unit variance, called autoscaling, is often appropriate for data where the variables have different units, such as process measurements, but is usually harmful for spectra, because it gives the flat noisy regions between bands the same weight as the bands themselves. The Savitzky-Golay filter illustrates the argument of this book in miniature. It is a purely mathematical operation, defined without any reference to chemistry. Yet every choice it demands, window width, polynomial degree, derivative order, depends on the widths of the bands, the level of noise and the kind of baseline in the particular spectra at hand. Used with that knowledge, it is one of the most effective tools available for separating chemical information from physical nuisance. Used by default, it can quietly erase the information it was meant to reveal. Hashtags: #Chemometrics #MultivariateCalibration #AnalyticalChemistry #SpectroscopicAnalysis #PrincipalComponentAnalysis #PCA #PartialLeastSquares #PLSRegression #InverseCalibration #SpectralPreprocessing #BaselineCorrection #ScatterCorrection #StandardNormalVariate #MultiplicativeScatterCorrection #SavitzkyGolayFilter #SpectralDerivatives #LatentVariables #PrincipalComponentRegression #NetAnalyteSignal #MultivariateRegression #CalibrationValidation #OutlierDetection #SpectralDataAnalysis #ModelTransfer #FutureOfChemometrics

Download the Book (PDF): Introduction A randomized trial shows that a school-based exercise programme lowers children's blood pressure. A cohort study finds that people with more years of education live longer. A policy evaluation reports that a cash transfer to poor households improves child growth. Each finding answers the question "does it work?" and immediately provokes a second one: how? Does exercise lower blood pressure because children lose weight, or because of something that happens to their blood vessels independent of weight? Does education extend life because educated people earn more, smoke less, or get better medical care? Does the cash transfer work because families buy more food, or because reduced financial stress changes how parents care for their children? Questions of this second kind are questions about mechanism, and the statistical tools built to answer them go by the name mediation analysis. The appeal of mediation analysis is obvious. A total effect tells you that pulling a lever moves an outcome. A decomposition into direct and indirect effects tells you which wires connect the lever to the outcome, and that knowledge is useful in ways a total effect is not. It tells a trial designer which component of a complex intervention to strengthen and which to drop. It tells a health system whether a disparity would shrink if an intermediate factor were equalized. It tells a scientist whether a theory about process survives contact with data. It can also rescue a failed trial from being simply a null result: if an intervention moved the mediator but the mediator did not move the outcome, the theory of change is wrong; if the intervention failed to move the mediator at all, the delivery was wrong. Those are very different lessons. The difficulty is that mechanism questions are harder than effect questions, and harder in a specific way that the older statistical literature concealed. For about two decades the dominant approach in psychology, management, education and much of public health was the "causal steps" procedure popularized by Reuben Baron and David Kenny in 1986. Regress the mediator on the exposure, regress the outcome on both, compare coefficients, and read the difference as the indirect effect. The procedure is easy, it runs in any regression package, and its paper became one of the most cited articles in the history of social science. It also rests on assumptions that its users rarely stated and often violated: that there is no confounding of the mediator–outcome relationship, that the exposure does not modify the effect of the mediator, and that the models are linear. When any of these fails, the coefficients can be combined in the prescribed way and produce a number that corresponds to no effect of any intervention one could imagine. Beginning with work by James Robins and Sander Greenland in 1992 and Judea Pearl in 2001, and developed through the following decade by Tyler VanderWeele, Stijn Vansteelandt, Kosuke Imai, Luke Keele, Dustin Tingley, Eric Tchetgen Tchetgen and others, a different approach took shape. It starts not from regression coefficients but from counterfactual definitions: what would the outcome have been had the exposure been set to one value and the mediator set to the value it would have taken under another? From those definitions come precise meanings for "direct" and "indirect" effects, formal conditions under which they can be estimated from data, estimators that work with binary outcomes, count mediators and interactions, and methods for asking how wrong the answer would be if the conditions failed. This counterfactual framework is now the standard in epidemiology and increasingly in psychology and political science, and it is embodied in software that any analyst can run. This booklet argues one thing throughout: a mediation estimate is only as credible as the untestable assumptions about the mediator that it requires, so the work of mediation analysis lies less in computing the decomposition than in defining the effect you want, defending the assumptions that identify it, and showing how the answer moves when those assumptions bend. Computation is the easy part. Modern software will produce a natural indirect effect with a bootstrap confidence interval in seconds. What the software cannot do is tell you whether the number means anything. That judgement depends on design, on subject-matter knowledge about what confounds the mediator and the outcome, and on honesty about the fact that randomizing the exposure does nothing to randomize the mediator. The chapters follow the logic of that argument. Chapter 1 sets out what a mediation question is, why it differs from an effect question, and when asking it is worthwhile. Chapter 2 examines the Baron and Kenny procedure and the product-of-coefficients method in detail: what they get right, where they break, and why a large literature built on them needs to be read with care. Chapter 3 introduces the counterfactual definitions of controlled direct effects, natural direct effects and natural indirect effects, and shows how they decompose a total effect. Chapter 4 turns to identification: the four no-confounding conditions, the peculiar "cross-world" assumption that natural effects require, and the problem of confounders that are themselves affected by the exposure. Chapter 5 covers estimation, from closed-form regression formulas through simulation, weighting, imputation and doubly robust methods, with attention to the software analysts actually use. Chapter 6 deals with nonlinearity: binary and count mediators and outcomes, exposure–mediator interaction, the failure of the product and difference methods outside linear models, odds ratios and the rare-outcome approximation, and the four-way decomposition that separates mediation from interaction. Chapter 7 is about sensitivity analysis for unmeasured mediator–outcome confounding, the single most important threat to any mediation claim. Chapter 8 extends the framework to multiple mediators, path-specific effects, interventional effects and time-varying settings, where the classical natural effects stop being identified and new estimands take their place. The Conclusion draws out what should change in the way mediation studies are designed, analysed and reported. A word on scope. The booklet assumes that you know what a regression model is, what a confidence interval means, and roughly what confounding is. It does not assume familiarity with potential outcomes notation, and it introduces every piece of notation in prose before using it. Formulas are written inline and kept short; where a derivation matters, the steps are given in words. Worked numerical examples use invented but realistic values chosen to make the arithmetic visible, and they are labelled as illustrations. Where the booklet describes published studies, it describes them as their authors reported them. The emphasis is on point exposures and single outcomes measured once, which covers most applied work; longitudinal mediation is treated in Chapter 8 in outline rather than in full, because its methods are still developing and require machinery beyond the scope of a short book. The intended reader is someone who has run, reviewed or relied on a mediation analysis and suspects the number was less solid than it looked: a trialist planning a mechanistic secondary analysis, an epidemiologist asked how much of a disparity runs through income, a psychologist whose reviewer asked for sensitivity analysis, a policy analyst asked to explain why a programme worked. For such readers the goal is not to add another technique but to change the order of operations. Decide first what effect you mean. Ask second what would have to be true for the data to reveal it. Only then compute, and afterwards report how fragile the result is. Done in that order, mediation analysis can deliver real insight into mechanism. Done in the old order, it produces confident decompositions of effects that may not exist. One further point deserves stating at the outset because it runs against a common intuition. Mediation analysis is not a way of extracting more from a randomized trial for free. The randomization that makes a trial's total effect credible protects only the comparison it randomizes. The moment an analysis splits the effect by a post-randomization variable, it becomes an observational study of that variable, with all the confounding problems observational studies carry. Many of the difficulties described in the chapters that follow stem from forgetting this. The mediator was not randomized. Everything else follows from taking that fact seriously. Chapter 1. What a Mediation Question Asks From "whether" to "how" Start with a simple picture. An exposure, call it A, affects an outcome, Y. A third variable, M, sits between them: A affects M, and M affects Y. A might be a drug, M the level of a biomarker, Y a clinical event. A might be a training programme, M the participant's confidence, Y whether they find work. A might be a genetic variant, M a behaviour such as smoking, Y a disease. In each case we believe that part of the effect of A on Y travels through M, and part may travel by other routes that do not involve M. The route through M is the indirect effect. Everything else is lumped together as the direct effect, where "direct" means only "not through M." It does not mean that the effect is biologically or psychologically immediate. A direct effect with respect to one mediator may itself be carried entirely by some other intermediate variable that the analysis has not measured. That qualification matters more than it appears. Direct and indirect effects are always defined relative to a chosen mediator. Change the mediator and the decomposition changes. An analysis that finds a large direct effect of education on mortality "not through income" has learned something about income, not about some mysterious unmediated pathway from schooling to survival. The direct effect is a residual category, and residual categories invite over-interpretation. Good mediation work keeps reminding its readers that the direct effect is the part not explained by the mediator studied, nothing more. The move from "whether" to "how" also changes what counts as a satisfactory answer. A total effect can be estimated without any theory of process: randomize the exposure, compare outcomes, done. A mediation question cannot be answered without a model of how the world works, because the answer depends on which variables confound the mediator and the outcome, on whether the exposure changes the way the mediator acts, and on what hypothetical manipulation of the mediator one has in mind. Mediation is where causal inference stops being a matter of design alone and starts to depend on substantive knowledge. Four reasons to ask It is worth being explicit about why one would want to decompose an effect, because different reasons point to different estimands, and much confusion in applied work comes from computing one quantity while wanting another. The first reason is explanation: to test a theory about process. A psychologist proposes that a mindfulness intervention reduces depressive relapse by reducing rumination. If the intervention reduces relapse and reduces rumination, and if the reduction in relapse is concentrated among those whose rumination falls, the theory gains support. If rumination does not move, or moves without any association with relapse, the theory is in trouble even if the intervention works. Here the scientific interest is in whether the pathway exists and how large a share of the effect it accounts for. The second reason is intervention design: to decide what to change. Suppose an antihypertensive drug reduces stroke, partly by lowering blood pressure and partly, perhaps, through effects on the vessel wall. A health system cannot give everyone the drug, but it can target blood pressure by other means. The relevant question then is what would happen to stroke if blood pressure were fixed at some target level, regardless of drug. That is a question about the effect of intervening on the mediator, and, as Chapter 3 shows, it corresponds most naturally to a controlled direct effect rather than to a natural indirect effect. The third reason is refining an intervention. Complex interventions such as a school programme with a curriculum, a teacher-training component and a parent outreach arm are expensive. If the effect runs mostly through one component's proximal target, the others may be dropped. Mediation analysis of trial data is routinely used for this purpose, and the Medical Research Council's guidance on process evaluation of complex interventions, published by Graham Moore and colleagues in the BMJ in 2015, explicitly recommends testing hypothesized mechanisms as part of evaluation. The fourth reason is understanding disparities. How much of the difference in infant mortality between two groups would remain if birth weight distributions were equalized? How much of the gender pay gap is attributable to occupation? Here the "exposure" is often not something anyone could manipulate, which raises its own conceptual problems, but the mediator usually is, at least in principle. The interest is in what a policy acting on the mediator would achieve. This purpose has given rise to its own literature on "disparity reduction" and "residual disparity" estimands, which are close relatives of the interventional effects discussed in Chapter 8. These four purposes are not interchangeable. Explanation asks about natural pathways as they operate in the population. Intervention design asks what would happen under a new manipulation of the mediator. The distinction between natural and controlled effects, which the older literature did not draw, is exactly the distinction between these purposes. An analyst who knows which purpose drives the study can choose the right estimand. One who does not will usually compute whatever the software reports by default. When the total effect is not enough, and when it is Mediation analysis is sometimes performed because a journal or reviewer expects it, not because the question demands it. It is worth asking, before starting, whether a decomposition adds anything. It adds little when the mediator is conceptually indistinguishable from the outcome. An intervention to increase physical activity that is "mediated by" step count is not being explained; step count is a measure of physical activity. It adds little when the mediator is measured at the same time as the outcome, because then the temporal ordering that mediation requires cannot be established and the "mediator" may equally be a consequence of the outcome. Cross-sectional mediation analysis, still common in survey research, has been criticized on exactly this ground by Scott Maxwell and David Cole, whose 2007 paper in Psychological Methods showed that cross-sectional estimates of longitudinal mediation can be severely biased in either direction, even when the underlying process is perfectly stable. It adds a great deal when the mediator is a plausible target for separate intervention, when the timing is clear, and when there is prior reason to believe the effect runs through more than one route. It also adds a great deal when the total effect is null. A null total effect is compatible with no effect at all, but it is also compatible with an indirect effect and a direct effect of opposite sign that cancel. The classic example, discussed at length in the epidemiological literature, is maternal smoking and infant mortality. Smoking lowers birth weight, and low birth weight is strongly associated with death, so smoking raises mortality in part through birth weight. Yet among low-birth-weight infants, those born to smokers have lower mortality than those born to non-smokers. This "birth weight paradox" is not a protective direct effect of smoking; Sonia Hernández-Díaz, Enrique Schisterman and Miguel Hernán showed in 2006 that it can be explained by conditioning on birth weight, a variable affected by smoking and sharing unmeasured causes (such as birth defects) with mortality. The example is a warning that conditioning on a mediator can create associations rather than reveal effects, a theme that recurs throughout this booklet. Mediation analysis also earns its place when the hypothesized mechanism is contested and the answer has scientific consequences. A good example comes from genetic epidemiology. Variants in the CHRNA5-CHRNA3-CHRNB4 nicotinic receptor gene cluster on chromosome 15q25.1 are associated with both heavier smoking and higher lung cancer risk. An obvious hypothesis is that the variants raise cancer risk only because carriers smoke more. Tyler VanderWeele and colleagues examined this question in a case-control study published in the American Journal of Epidemiology in 2012, using counterfactual mediation methods with cigarettes per day as the mediator. They found that only a small fraction of the association between the variants and lung cancer operated through the number of cigarettes smoked, and that most of it appeared to operate by other routes, perhaps through depth of inhalation or a direct effect on lung tissue. That conclusion is only as good as the measurement of smoking and the control of confounders of smoking and cancer, and the authors were careful about both, but it illustrates a case where the decomposition changed the scientific question being asked. Mediation, confounding and moderation are different things Three roles that a third variable can play are often confused, and the confusion is costly. A confounder is a common cause of exposure and outcome. It lies before the exposure in time and produces a spurious association between them. Adjusting for a confounder removes bias. Age confounds the relation between coffee drinking and heart disease because older people drink different amounts of coffee and have more heart disease. A mediator lies on a causal path from exposure to outcome. It comes after the exposure and before the outcome. Adjusting for a mediator in the usual way does not remove bias in the total effect; it removes part of the effect itself, and, if the mediator shares unmeasured causes with the outcome, it introduces new bias. Blood pressure mediates part of the effect of salt intake on stroke. A moderator (or effect modifier) is a variable across whose levels the effect of the exposure differs. It need not lie on any causal path. The effect of a smoking-cessation drug may differ by sex without sex being affected by the drug. The statistical signatures of these roles can be identical. In a regression of outcome on exposure and a third variable, the coefficient on the exposure changes when the third variable is added whether that variable is a confounder or a mediator. No statistical test can distinguish the two; the distinction is causal and temporal, and it must come from knowledge of the subject. David MacKinnon, Jennifer Krull and Chondra Lockwood made this point formally in a 2000 paper in Prevention Science, showing that mediation, confounding and suppression are statistically equivalent in the linear model. This is the first of many places where mediation analysis depends on assumptions the data cannot check. The roles can also combine. A variable can be both a mediator and a moderator: the exposure can change the level of the mediator and also change how strongly the mediator affects the outcome. That combination, exposure–mediator interaction, is the case in which traditional methods fail most visibly, and it is treated in Chapters 3 and 6. A variable can also be a confounder of the mediator–outcome relation while being affected by the exposure, which, as Chapter 4 explains, makes the classical natural effects unidentifiable without further assumptions. The minimum causal structure It helps to fix a standard picture in words, because nearly every argument in the rest of the booklet refers back to it. There are four kinds of variable. The exposure A. The mediator M, which follows A. The outcome Y, which follows M. And three kinds of confounders: C1, common causes of A and Y; C2, common causes of M and Y; and C3, common causes of A and M. In the simplest case all of these are measured before the exposure, and none is affected by it. A fifth kind, L, is a common cause of M and Y that is itself affected by A. Its presence changes the analysis fundamentally. Arrows run from A to M, from M to Y, and from A to Y directly. Arrows run from each confounder to the two variables it confounds. Any unmeasured variable with arrows into both M and Y is the principal enemy of mediation analysis, and it is precisely the variable that randomizing A does nothing to remove. In a trial, C1 and C3 are balanced by randomization. C2 is not. The trialist who reports a mediated effect without adjusting for, or at least discussing, common causes of mediator and outcome has run an observational analysis and presented it as experimental. A concrete case makes this vivid. In a trial of a weight-loss programme with the outcome of blood pressure at 12 months and the mediator of weight change at 6 months, randomization balances everything at baseline. But among participants in either arm, those who lost more weight may differ from those who lost less in ways that also affect blood pressure: they may have changed their salt intake, their alcohol consumption, their sleep, their adherence to antihypertensive medication. Each of these is a potential mediator–outcome confounder, and if unmeasured, each will bias the estimated indirect effect of the programme through weight. The size and direction of that bias is the subject of Chapter 7. What a good mediation question looks like Pulling these threads together, a well-formed mediation question has five features. It names the exposure, mediator and outcome with their timing, and the timing is such that the mediator is measured after the exposure and before the outcome, ideally with a gap sufficient for effects to operate. It states the purpose: explanation of natural pathways, prediction of the effect of a mediator intervention, refinement of a multi-component intervention, or disparity reduction. The purpose determines the estimand. It identifies, before analysis, the variables believed to confound each of the three relations: exposure–outcome, exposure–mediator and mediator–outcome. In a trial, the third list is the one that requires most thought. It considers whether the exposure plausibly modifies the effect of the mediator, and plans a model that allows for it. It plans a sensitivity analysis for the confounding that remains after adjustment, with a view about what magnitude of unmeasured confounding would be plausible. An example of a question with all five features: in a randomized trial of a smartphone-delivered cognitive behavioural therapy programme for insomnia, what proportion of the effect on depressive symptoms at 24 weeks operates through improvement in insomnia severity at 12 weeks, in the natural course of the trial population? Mediator–outcome confounders measured at baseline include prior depression history, anxiety, alcohol use and shift work; the analysis allows the effect of insomnia improvement on depression to differ by arm; and the sensitivity analysis asks how strong an unmeasured common cause of insomnia improvement and depression change would have to be to reduce the indirect effect to zero. Questions of roughly this form have been asked in large digital insomnia trials, and they illustrate the value of the approach: a finding that most of the effect runs through sleep has implications for how such programmes are designed and for theories linking sleep and mood. Contrast that with a question lacking these features: does self-esteem mediate the relation between social media use and depressive symptoms in a cross-sectional survey of adolescents? The temporal order is unknown, the three variables may all be consequences of a common underlying disposition, reverse causation is likely, and no purpose beyond "understanding" has been specified. A number can be computed. It will be reported with a confidence interval. It has no clear interpretation. The rest of the argument Everything that follows elaborates on a simple asymmetry. Estimating a total effect in a randomized trial requires only that the randomization was done properly. Estimating how that effect divides between pathways requires, in addition, that we know and have measured every common cause of mediator and outcome, that the models relating them are correct or flexible enough, and, for the natural effects most analysts report, an assumption that can never be tested even in principle. These requirements are not reasons to abandon mediation analysis. They are reasons to do it deliberately: to be clear about the estimand, to put effort into measuring mediator–outcome confounders, to use estimators that do not break under nonlinearity, and to report how the conclusions depend on what could not be measured. The older approach did none of these things, not because its authors were careless but because the framework in which it was developed did not make the assumptions visible. Chapter 2 examines that framework closely, because much of the published mediation literature still uses it, and reading that literature critically requires knowing exactly where it fails. Chapter 2. The Causal Steps Tradition and Where It Breaks The procedure and its appeal In 1981 Charles Judd and David Kenny published a paper in Evaluation Review called "Process analysis," which laid out a regression-based strategy for examining how treatments in evaluation research produce their effects. Five years later Reuben Baron and David Kenny, writing in the Journal of Personality and Social Psychology, set out the distinction between moderator and mediator variables and codified what became known as the causal steps approach. The 1986 paper has been cited well over one hundred thousand times. For most of the following quarter century, to "test mediation" in psychology, organizational behaviour, marketing, education and much of health research meant to follow its steps. The procedure is easy to state in terms of three linear regressions. Let A be the exposure, M the mediator and Y the outcome. First, regress Y on A; call the coefficient c, the total effect. Second, regress M on A; call the coefficient a. Third, regress Y on both A and M; call the coefficient on M b and the coefficient on A c-prime, written here as c'. Mediation is claimed if four conditions hold: c is statistically significant, a is significant, b is significant, and c' is smaller in absolute value than c. If c' is no longer significant, the mediation is called "full" or "complete"; if it is reduced but still significant, the mediation is "partial." The size of the indirect effect is estimated as the product a times b, or equivalently in ordinary least squares with the same sample as the difference c minus c'. In linear models fitted to the same observations these two quantities are exactly equal, a fact that gave the approach an attractive internal coherence. To test whether the indirect effect differs from zero, Michael Sobel's 1982 delta-method formula gave a standard error for the product ab, the square root of b squared times the variance of a plus a squared times the variance of b. Dividing ab by this standard error yields a z statistic. The Sobel test became a standard companion to the causal steps. The appeal is easy to understand. The steps map onto a verbal theory: A affects M, M affects Y, and once M is accounted for, A matters less. They require nothing beyond regression. They produce a clear verdict. They could be taught in an afternoon. They also contain a genuine insight that survives in the modern literature: in a linear system without interaction and without confounding, the indirect effect really is the product of the path coefficients, and the direct effect really is the coefficient on the exposure after holding the mediator fixed. The trouble lies entirely in the qualifying phrase. Table 1 summarizes each step, what it was meant to establish, and the main ways it fails. Table 1. The causal steps procedure and its principal failures. Step Regression Intended inference Principal failure 1 Y on A An effect exists to be mediated Opposing pathways can cancel; low power 2 M on A A changes the mediator Usually sound if A is randomized 3 Y on A and M M affects Y holding A fixed Unmeasured M–Y confounding biases b 4 Compare c' with c Share of effect through M Assumes no A–M interaction, linearity Test Sobel z for ab Indirect effect differs from zero Product not normal; low power Step one: requiring a total effect The first step demands a significant total effect before mediation is examined. The reasoning was that there must be an effect before one can ask how it is transmitted. The reasoning fails in two ways. The first is substantive. A total effect near zero can conceal two genuine pathways of opposite sign. Consider a job-training programme that raises participants' skills, which improves employment, but also keeps them in training during months they would otherwise have spent searching for work, which reduces employment in the short term. The total effect at six months may be zero while both the indirect effect through skills and the direct "lock-in" effect are real. Requiring step one would forbid any look at the mechanism. This pattern, sometimes called inconsistent mediation or suppression, was discussed by MacKinnon, Krull and Lockwood in 2000 and is not rare. The second failure is statistical. The test of the total effect can have less power than a test of the indirect effect. Xinshu Zhao, John Lynch and Qimei Chen, in a widely read 2010 paper in the Journal of Consumer Research titled "Reconsidering Baron and Kenny: Myths and truths about mediation analysis," argued that the only requirement for demonstrating mediation is a significant indirect effect, and that step one should be dropped. David Kenny and Charles Judd themselves, in a 2014 paper in Psychological Science on "power anomalies in testing mediation," showed that when the direct effect is small, a test of the indirect effect ab can have far more power than a test of the total effect c, so that a study can be adequately powered to detect mediation and still fail step one. By the mid-2010s most methodologists had abandoned the requirement, although it still appears in applied papers and reviewer comments. Steps three and four: the hidden assumption The fatal problem lies in step three. The coefficient b, obtained by regressing Y on A and M, is interpreted as the causal effect of M on Y holding A fixed. That interpretation is valid only if there are no unmeasured common causes of M and Y. The exposure may be randomized, but the mediator almost never is. People differ in their mediator values for reasons other than the exposure, and those reasons frequently also affect the outcome. Baron and Kenny were not unaware of this. Judd and Kenny's 1981 paper discussed specification error and the possibility that the mediator is caused by the outcome, and the 1986 paper discussed measurement error. But the causal steps as practised treated b as a causal effect by default, and the procedure contained no step at which mediator–outcome confounding was addressed. John Bullock, Donald Green and Shang Ha put the point bluntly in a 2010 paper in the Journal of Personality and Social Psychology titled "Yes, but what's the mechanism? (don't expect an easy answer)": the conventional approach "is biased" whenever the mediator is not randomly assigned, and randomizing the exposure does not help. A worked example shows how large the bias can be. Suppose A is a randomized binary treatment. The true data-generating process is linear: M equals 0.5 times A plus U plus independent noise with variance 1, and Y equals 0.3 times A plus 0.4 times M plus U plus independent noise. Here U is an unmeasured characteristic, standardized to variance 1, that raises both M and Y. The true indirect effect is 0.5 times 0.4, or 0.2. The true direct effect is 0.3. The total effect is 0.5. Now run the causal steps. The regression of Y on A recovers the total effect, 0.5, because A is randomized. The regression of M on A recovers a equal to 0.5, again because of randomization. But the regression of Y on A and M does not recover b. Conditional on A, the variance of M is 2 (1 from U and 1 from noise), and its covariance with U is 1. The regression therefore attributes to M part of the effect of U on Y: the estimated b is 0.4 plus 1 divided by 2, which is 0.9. Because the total effect is fixed at 0.5 and the product ab is now 0.5 times 0.9, or 0.45, the estimated direct effect c' shrinks to 0.05. The causal steps declare complete mediation: a significant a, a significant b, and a direct effect indistinguishable from zero. In truth 60 per cent of the effect does not run through M at all. The analyst has confidently reported a mechanism that accounts for 90 per cent of the effect when it accounts for 40 per cent. The bias can run the other way. If U raised M but lowered Y, the estimated b would be smaller than 0.4, perhaps negative, and the analysis would miss a real mechanism. Unmeasured mediator–outcome confounding does not merely add noise; it distorts both the indirect and the direct effect, in opposite directions, because their sum is pinned by the unbiased total effect. Randomized trials offer no protection. Every one of the numbers above came from an analysis in which A was randomized. This is the most important and least appreciated lesson of the modern mediation literature, and it applies to every mediation analysis of every trial. Measurement error in the mediator A related problem arises when the mediator is measured with error. Suppose there is no confounding, but the mediator is a noisy psychological scale or a single blood pressure reading. Classical measurement error in M attenuates b toward zero. The regression of Y on A and the error-prone M then attributes to A some of the effect that truly runs through M. Using the same true values as before but without U, suppose the reliability of M conditional on A is 0.6, meaning that 60 per cent of the residual variance in the measured mediator reflects the true mediator. The estimated b falls to about 0.6 times 0.4, or 0.24. The estimated indirect effect becomes 0.5 times 0.24, or 0.12. Because the total effect is still 0.5, the estimated direct effect rises to 0.38. Measurement error has made a real mechanism look weaker and has inflated the residual "direct" effect. Baron and Kenny noted this in 1986 and suggested latent variable models as a remedy. Saskia le Cessie and colleagues gave a detailed treatment for epidemiological settings in a 2012 paper in Epidemiology, showing that the direction of bias in direct effects depends on the type of measurement error and whether the mediator is continuous or categorical. Measurement error and confounding interact in awkward ways. A mediator that is both confounded and noisy can yield an estimate that happens to be close to the truth for the wrong reasons, or badly wrong in either direction. There is no substitute for measuring the mediator well and thinking hard about what causes it. Interaction and nonlinearity The causal steps assume, without stating it, that the effect of M on Y is the same whether A is 0 or 1. If the exposure modifies the effect of the mediator, there is no single b, and the product ab no longer corresponds to any well-defined effect. Suppose a drug lowers a biomarker, and the biomarker matters for the outcome only among treated patients because the drug and the biomarker act together. Then the "effect of M holding A fixed" depends on where A is held, and the causal steps have no way to say which value to use. Linda Valeri and Tyler VanderWeele, in a 2013 paper in Psychological Methods, showed how the counterfactual definitions of Chapter 3 resolve this by making explicit which level of the exposure is used for each part of the decomposition. The causal steps, by contrast, typically omit the interaction term altogether, forcing a constant b and producing biased estimates when the true effect varies. The causal steps also assume linear models throughout. When the outcome is binary and modelled with logistic regression, the coefficients are log odds ratios, and the odds ratio is non-collapsible: adding a variable to a logistic model changes the coefficient on the exposure even when that variable is not a confounder or mediator at all, simply because of the mathematics of the logit scale. The difference method, c minus c', therefore confounds mediation with non-collapsibility. The product method, ab, multiplies a coefficient on one scale by a coefficient on another. The two methods no longer agree, and neither corresponds to a causal effect in general. Chapter 6 treats this problem in detail and shows how the counterfactual framework produces valid estimates in its place. Inference: from Sobel to the bootstrap Even where the point estimate ab is valid, the Sobel test for it has poor properties. The product of two normally distributed estimates is not normally distributed; its sampling distribution is skewed, and more so when the true effects are small. A symmetric confidence interval based on the Sobel standard error is therefore mis-centred, and the test has low power. David MacKinnon and colleagues compared fourteen methods for testing intervening variable effects in a 2002 simulation study in Psychological Methods and found that the causal steps and the Sobel test had low power and inaccurate Type I error rates in many realistic conditions. Two remedies emerged. The first uses the exact distribution of the product of two normal variables, implemented for instance in the RMediation package for R developed by Davood Tofighi and MacKinnon. The second, far more widely adopted, is the bootstrap. Resample the data with replacement many times, re-estimate a and b in each resample, compute the product, and take the 2.5th and 97.5th percentiles of the resulting distribution as a 95 per cent confidence interval. Kristopher Preacher and Andrew Hayes promoted this approach in papers published in 2004 and 2008 in Behavior Research Methods, and Hayes's PROCESS macro for SPSS, SAS and R made it standard practice in psychology and business research. Hayes's textbook, Introduction to Mediation, Moderation, and Conditional Process Analysis, now in its third edition (2022), is the most widely used applied guide in those fields. The bootstrap solved an inferential problem. It did not solve the identification problem. A bootstrapped confidence interval around a product of coefficients that is biased by mediator–outcome confounding is a precise interval around the wrong number. Much of the applied literature moved from Sobel to bootstrap and considered the question of mediation settled, when the more serious threats had not been addressed at all. Full and partial mediation, and other artefacts The distinction between full and partial mediation depends on whether c' is statistically significant, and significance depends on sample size. A small study will declare full mediation more often than a large one studying the same process, simply because it lacks the power to detect a residual direct effect. Derek Rucker, Kristopher Preacher, Zakary Tormala and Richard Petty made this point in a 2011 paper in Social and Personality Psychology Compass, recommending that researchers abandon the full–partial language and report effect sizes instead. The concept of full mediation also invites a conceptual error: the belief that once a mediator fully accounts for an effect, the mechanism has been identified. But a direct effect of zero relative to one mediator says nothing about what lies between A and M, or between M and Y. Every mechanism is itself made of further mechanisms. A related artefact is the "proportion mediated," often computed as ab divided by c. When c is small, this ratio is unstable and can exceed 1 or be negative. When the direct and indirect effects have opposite signs, it has no sensible interpretation. The proportion mediated is useful when both effects share a sign and the total effect is clearly non-zero; otherwise reporting both effects separately is better. What survives It would be wrong to conclude that the causal steps literature is worthless. In settings where the exposure is randomized, the models are genuinely linear, the exposure does not modify the mediator's effect, and the mediator–outcome relation is unconfounded or well adjusted, the product-of-coefficients estimate coincides with the natural indirect effect defined in Chapter 3. The modern framework does not reject the older estimates; it shows under exactly what conditions they are valid and supplies alternatives when those conditions fail. What the older tradition lacked was a language in which the conditions could be stated. It spoke of variables and coefficients rather than of interventions and potential outcomes. It could not say what "the effect of M holding A fixed" meant when the effect of M depended on A, or what "the indirect effect" meant when the outcome was binary. The counterfactual framework provides that language. Its first contribution is not a new estimator but a set of definitions, which is where Chapter 3 begins. For a reader of the published literature, the practical upshot is a short checklist. Was the mediator measured after the exposure and before the outcome? Were mediator–outcome confounders named and adjusted for, and if the exposure was randomized, did the authors recognize that the mediator was not? Did the model allow for exposure–mediator interaction? If the outcome was binary, how were the coefficients combined? Was there any sensitivity analysis? A study that answers all five well may be trustworthy whatever method it used. A study that answers none of them, however elegant its bootstrap intervals, offers a mechanism claim resting on assumptions it never stated. Chapter 3. Counterfactual Definitions of Direct and Indirect Effects Potential outcomes with a mediator The counterfactual approach to causation defines an effect by comparing outcomes under different hypothetical interventions on the same unit. For a binary exposure A, each person has two potential outcomes: Y(1), the outcome that would occur if the exposure were set to 1, and Y(0), the outcome if it were set to 0. Only one is ever observed. The average causal effect is the mean of Y(1) minus Y(0) across the population, and it can be estimated from data when the exposure is randomized, or when all common causes of exposure and outcome are measured. Mediation adds a second layer. The mediator, too, has potential values: M(1) is the value it would take if the exposure were set to 1, M(0) the value if the exposure were set to 0. And the outcome now has potential values indexed by both the exposure and the mediator: Y(a, m) is the outcome that would occur if the exposure were set to a and the mediator were set to m. With a binary exposure and a mediator taking many values, each person has a whole surface of potential outcomes, one for each combination of exposure and mediator level. Two further assumptions connect these objects to each other and to the data. Consistency says that if a person actually received exposure a and actually had mediator value m, their observed outcome equals Y(a, m); similarly, their observed mediator equals M(a). Composition says that Y(a), the outcome under exposure a alone, equals Y(a, M(a)): setting the exposure to a and letting the mediator take whatever value it naturally would under a yields the same outcome as setting the exposure alone. These conditions sound trivial but carry content, as the discussion of well-defined interventions below makes clear. With this notation, effects of different kinds can be written as contrasts between potential outcomes. James Robins and Sander Greenland, in a 1992 paper in Epidemiology called "Identifiability and exchangeability for direct and indirect effects," and Judea Pearl, in a 2001 paper at the Conference on Uncertainty in Artificial Intelligence titled "Direct and indirect effects," set out the definitions that the rest of this booklet uses. Table 2 collects them. Table 2. Counterfactual effects used in mediation analysis, binary exposure. Effect Contrast of mean potential outcomes Plain meaning Total effect (TE) Y(1) versus Y(0) Effect of exposure through all routes Controlled direct effect, CDE(m) Y(1, m) versus Y(0, m) Effect of exposure if mediator fixed at m for all Natural direct effect (NDE) Y(1, M(0)) versus Y(0, M(0)) Effect of exposure with mediator at its unexposed level Natural indirect effect (NIE) Y(1, M(1)) versus Y(1, M(0)) Effect of shifting mediator from unexposed to exposed level, exposure held at 1 Portion eliminated, PE(m) TE minus CDE(m) Effect removed by fixing the mediator at m The controlled direct effect The controlled direct effect at mediator level m is the average of Y(1, m) minus Y(0, m). It answers the question: if we could set everyone's mediator to the same value m, what effect would the exposure have? It is the most intuitive of the direct effects, and it corresponds exactly to an imagined experiment in which both exposure and mediator are randomized, or to a policy that fixes the mediator for everyone. Controlled direct effects have two important properties. First, they depend on m. If the exposure interacts with the mediator, the controlled direct effect differs at each mediator level, and there are as many controlled direct effects as there are values of m. In a study of whether an antiretroviral regimen reduces mortality other than by suppressing viral load, the controlled direct effect with viral load fixed at "undetectable" may be quite different from the controlled direct effect with viral load fixed at a high level. Second, there is no corresponding "controlled indirect effect." The difference between the total effect and a controlled direct effect, which VanderWeele calls the portion eliminated, measures how much of the exposure's effect would be removed by fixing the mediator at m. That is a useful policy quantity, but it is not an indirect effect in the sense of a pathway, because it mixes the mediated effect with any interaction between exposure and mediator. When there is no interaction, the portion eliminated equals the natural indirect effect; when there is interaction, it does not. The controlled direct effect is the right estimand when the question is about intervening on the mediator. If a health authority wants to know what the effect of a social disadvantage on child development would be if every child attended high-quality preschool, that is a controlled direct effect with the mediator (preschool attendance) set to 1. It does not require the difficult assumptions that natural effects require, and it can be estimated under assumptions no stronger than those for the joint effect of two exposures. Natural direct and indirect effects Natural effects answer a different question. The natural direct effect is the average of Y(1, M(0)) minus Y(0, M(0)). In words: change the exposure from 0 to 1 but hold each person's mediator at the value it would have taken under exposure 0. The mediator is not fixed at a common value; it is fixed at each person's own "natural" unexposed level. The natural indirect effect is the average of Y(1, M(1)) minus Y(1, M(0)): hold the exposure at 1, and shift each person's mediator from what it would have been unexposed to what it would be exposed. The virtue of natural effects is that they decompose the total effect exactly, whatever the form of the model and whatever interactions exist. Adding the two contrasts, the term Y(1, M(0)) cancels, leaving Y(1, M(1)) minus Y(0, M(0)), which by composition equals Y(1) minus Y(0), the total effect. So TE = NDE + NIE on the difference scale, always. On the ratio scale, the decomposition is multiplicative: the total effect risk ratio equals the natural direct effect risk ratio times the natural indirect effect risk ratio. This is exactly the property that the causal steps sought and could not provide outside linear models. Natural effects also correspond to the explanatory purpose described in Chapter 1. They describe how the effect of the exposure actually divides between pathways as the mediator varies naturally across the population, rather than what would happen under a new intervention setting everyone's mediator to the same value. When a scientist asks "how much of the effect of this programme runs through self-efficacy?" the natural indirect effect is usually what is meant. The cost is a peculiar kind of counterfactual. The quantity Y(1, M(0)) combines two incompatible worlds: in one, the person is exposed; in the other, they are unexposed and their mediator takes the value M(0). No experiment can observe both at once, even in principle, because a single person cannot simultaneously be exposed and unexposed. For this reason Y(1, M(0)) is sometimes called a cross-world counterfactual. As Chapter 4 explains, this makes natural effects identifiable only under an assumption that cannot be verified by any experiment, not merely an assumption we happen to lack data for. Thomas Richardson and James Robins have developed a detailed critique along these lines, and some methodologists prefer to avoid natural effects for this reason. Others, including Pearl and VanderWeele, regard them as meaningful descriptions of mechanism whose identifying assumption is untestable but interpretable. The practical consequence is that natural effects require more caution than controlled ones, and the interventional analogues described in Chapter 8 exist partly to relieve this difficulty. Two decompositions, and why they differ The decomposition above is not the only one. One can equally write the total effect as the sum of Y(1, M(1)) minus Y(0, M(1)), a direct effect with the mediator held at its exposed level, and Y(0, M(1)) minus Y(0, M(0)), an indirect effect with the exposure held at 0. Robins and Greenland called the first pair the pure direct effect and total indirect effect; the second, the total direct effect and pure indirect effect. The word "pure" signals that the contrast is taken at the reference exposure level; "total" signals that it includes any interaction between exposure and mediator. When the exposure does not modify the effect of the mediator, the pure and total versions coincide, and there is only one natural direct and one natural indirect effect. When there is interaction, they differ, and the difference between the total indirect effect and the pure indirect effect is exactly the mediated interaction, the part of the effect that requires both the exposure to change the mediator and the exposure to modify the mediator's effect. Chapter 6 shows how VanderWeele's four-way decomposition separates this piece explicitly. A worked example makes the distinctions concrete. Suppose the mediator is continuous and follows the linear model M = 2 + 1 times A, so the exposure raises the mediator by one unit from a baseline of 2. The outcome follows Y = 0.5 A + 0.3 M + 0.4 A M, with an exposure–mediator interaction coefficient of 0.4. Assume no confounding and ignore noise, which averages out. The total effect is the mean of Y under A = 1 with M = 3, minus Y under A = 0 with M = 2. That is (0.5 + 0.9 + 1.2) minus 0.6, which equals 2.0. The pure natural direct effect holds M at its unexposed level of 2 and changes A: 0.5 plus 0.4 times 2, which equals 1.3. The total natural indirect effect holds A at 1 and shifts M from 2 to 3: the effect of one unit of M when A = 1 is 0.3 plus 0.4, or 0.7. The sum, 1.3 plus 0.7, recovers the total effect of 2.0. The alternative decomposition gives a total natural direct effect of 0.5 plus 0.4 times 3, or 1.7, holding M at its exposed level of 3, and a pure natural indirect effect of 0.3 times 1, or 0.3, shifting M with A held at 0. Again 1.7 plus 0.3 equals 2.0. So the "proportion mediated" is either 35 per cent (0.7 of 2.0) or 15 per cent (0.3 of 2.0), depending on which decomposition is used. Neither is wrong. They answer slightly different questions, and the gap between them, 0.4, is the mediated interaction. The controlled direct effects, meanwhile, are 0.5 plus 0.4 m: 0.5 at m = 0, 1.3 at m = 2, 2.1 at m = 4. A causal steps analysis that omitted the interaction term would fit a single slope for M and a single coefficient for A, and would report a single indirect effect corresponding to none of these quantities. The convention in most software, following Valeri and VanderWeele, is to report the pure natural direct effect and the total natural indirect effect, so that the indirect effect includes the mediated interaction. But analysts should know which convention their software uses and state it, because in the presence of interaction the choice changes the answer materially. Well-defined interventions on the mediator All the effects in Table 2 other than the total effect involve setting the mediator to some value, either a fixed m or a person's own M(0). For these quantities to be meaningful, it must make sense to imagine intervening on the mediator, and the result of that intervention must not depend on how it was done. For some mediators this is unproblematic. A biomarker that can be altered by a drug, a behaviour that can be prompted by a reminder, a service that can be provided or withheld: each can be manipulated in reasonably well-defined ways. For others it is not. Body mass index can be changed by diet, by exercise, by surgery, by illness, or by drugs, and these routes have different consequences for health. Miguel Hernán and Sarah Taubman, in a 2008 paper in the International Journal of Obesity titled "Does obesity shorten life? The importance of well-defined interventions to answer causal questions," argued that without specifying the intervention, the effect of changing BMI is ill-defined, and so is any mediated effect through BMI. The same applies to mediators such as "stress," "self-esteem" or "social cohesion," which may be changed in many ways with different downstream effects. This does not make mediation analysis with such mediators pointless, but it does make the interpretation more modest. The natural indirect effect through BMI describes how much of an exposure's effect is associated with the change in BMI it produces, under the assumption that the way the exposure changes BMI is the relevant version of the intervention. That is a reasonable thing to estimate, as long as it is not read as a promise about what a weight-loss programme would achieve. Exposures raise similar issues. Mediation analyses of race, sex or social class, now common in disparities research, treat characteristics that cannot be manipulated as the exposure. The counterfactual Y(a) is then difficult to define. One response, developed by Tyler VanderWeele and Whitney Robinson in a 2014 paper in Epidemiology, is to reinterpret such analyses as describing what would happen to a disparity if the distribution of a modifiable mediator were changed, without claiming a causal effect of the non-manipulable characteristic itself. That reframing moves the analysis toward interventional estimands, discussed in Chapter 8. Choosing among the estimands Returning to the purposes set out in Chapter 1, a rough mapping emerges. For intervention design and policy questions about fixing the mediator, the controlled direct effect and the portion eliminated are the natural estimands, and they require weaker assumptions. For explanation and theory testing, the natural direct and indirect effects are the natural estimands, with the cost that they require a cross-world assumption. For refining multi-component interventions, either may be relevant: if the question is whether the effect would survive removal of a component that acts on M, a controlled effect is appropriate; if the question is how much of the current effect operates through M, a natural effect is. For disparities, interventional effects are often the best fit. Ashley Naimi, Jay Kaufman and Richard MacLehose argued in a 2014 paper in the International Journal of Epidemiology, "Mediation misgivings," that natural effects are often poorly suited to clinical and public health questions because no feasible intervention corresponds to them. Trang Quynh Nguyen, Ian Schmid and Elizabeth Stuart, in a 2021 paper in Psychological Methods, set out a more detailed framework for choosing among controlled, natural, interventional and other estimands according to what the researcher wants to learn. Both papers make the same underlying point: the estimand should be chosen before the software is opened, and the choice should follow from the question. A last observation concerns scale. The effects in Table 2 are defined as differences in means, which for a binary outcome are risk differences. They can equally be defined as ratios. On the risk-ratio or odds-ratio scale, the decomposition is multiplicative rather than additive, and the proportion mediated is computed differently. For a binary outcome, the proportion mediated on the ratio scale is conventionally computed as RRNDE times (RRNIE minus 1), divided by RR_TE minus 1, which approximates the proportion on the risk-difference scale. The choice of scale matters for interaction and for the proportion mediated, and Chapter 6 returns to it. With these definitions in place, the next question is what would have to be true for the data to reveal them. That is the subject of identification. Hashtags: #CausalMediationAnalysis #MediationAnalysis #CausalInference #DirectEffects #IndirectEffects #TotalEffect #ControlledDirectEffect #NaturalDirectEffect #NaturalIndirectEffect #PotentialOutcomes #CounterfactualMediation #MediatorOutcomeConfounding #ExposureMediatorInteraction #CausalSteps #BaronAndKenny #ProductOfCoefficients #BootstrapMediation #SensitivityAnalysis #FourWayDecomposition #PathSpecificEffects #InterventionalEffects #MultipleMediators #LongitudinalMediation #MechanismAnalysis #FutureOfCausalMediation

Download the Book (PDF): Introduction A biofilm experiment rarely fails loudly. The pump runs, the channels fill with a pale film, the confocal stack looks handsome, and a spreadsheet eventually reports a biovolume or a log reduction to two decimal places. The failure, when it comes, arrives later: a colleague in another building cannot reproduce the effect, a coating that cleared every well of a microtiter plate is fouled within a fortnight in a cooling tower, a compound that "eradicated" a biofilm at 64 micrograms per millilitre does nothing to the infection on a prosthetic joint. In almost every such case the organism behaved exactly as organisms do. What went wrong was that the experiment measured something other than what its authors believed it measured. This booklet is built on a single conviction: a biofilm result is a property of the measurement system as much as of the microbe. The flow regime decides what kind of community grows. The stain decides which cells are called alive. The microscope decides which part of the community is seen at all, and the thresholding algorithm decides which of the seen voxels count as biomass. The disinfectant test decides whether "kill" means a reduction in colony counts, a fall in fluorescence, or the absence of regrowth on a peg. None of these choices is neutral, and none of them can be repaired after the fact. A laboratory that treats its biofilm protocols as instruments, designed deliberately, calibrated with controls, and reported in enough detail to be rebuilt elsewhere, will produce results that survive contact with other laboratories and with the real surfaces the work is meant to inform. Why biofilms demand their own methods The idea that most bacteria in nature live attached to surfaces, embedded in a self-produced matrix of polysaccharides, proteins, lipids and extracellular DNA, is now old enough to be taken for granted. J. William Costerton and colleagues made the case forcefully in the 1990s, and by 1999 the argument that biofilms underlie a large share of persistent infections had reached the pages of Science. Hans-Curt Flemming and Jost Wingender's review of the matrix, published in 2010, gave the field a vocabulary for the "house of the biofilm cells", and later syntheses described biofilms as an emergent form of bacterial life, with properties such as nutrient retention, protection from desiccation and tolerance to antimicrobials that do not belong to any individual cell. Those emergent properties are exactly why biofilm methods cannot be borrowed wholesale from planktonic microbiology. A minimum inhibitory concentration measured on a turbid broth culture describes cells that are suspended, evenly exposed to the drug, and mostly growing. A biofilm contains gradients of oxygen, nutrients, pH and signalling molecules across tens of micrometres; cells near the substratum may be starved and barely dividing while cells at the surface grow briskly. A matrix that binds cationic antimicrobials, a subpopulation of dormant persister cells, and slow penetration of reactive biocides together explain why biofilm-grown bacteria routinely survive concentrations far above the planktonic MIC. Measuring that tolerance requires growing the community under defined conditions, exposing it in a way that respects its structure, and then recovering or imaging the survivors without introducing artefacts larger than the effect. What this booklet covers The chapters follow the life of an experiment. Chapter 1 frames the problem of measurement: what a biofilm endpoint is, which laboratory models exist, what each is good for, and why standardized methods from ASTM International and reporting guidelines such as MIABiE matter even in academic work. Chapter 2 supplies the fluid mechanics a biologist needs to set a flow rate on purpose rather than by habit, with worked calculations of wall shear stress, Reynolds number and residence time for real devices. Chapter 3 is a practical guide to building, sterilizing, inoculating and running flow cells and microfluidic channels, including the failures that most often ruin a week's work. Chapter 4 deals with surfaces, biological and industrial: conditioning films, host-derived coatings, medical polymers, stainless steel, membranes and the problem of matching laboratory substrata to the surfaces that matter. The second half concerns seeing and counting. Chapter 5 examines LIVE/DEAD staining with SYTO 9 and propidium iodide, the most widely used and most widely misread viability assay in biofilm research, and sets it alongside the alternatives. Chapter 6 covers confocal acquisition: objectives, sampling, laser power, spectral separation and the statistics of where to point the microscope. Chapter 7 turns image stacks into numbers with COMSTAT2, BiofilmQ and related tools, and places image-derived biomass beside bulk measures such as crystal violet, resazurin and viable counts. Chapter 8 addresses disruption and antimicrobial testing, from the Calgary Biofilm Device to the CDC Biofilm Reactor and the single tube method that underpins United States Environmental Protection Agency biofilm claims, and treats dispersal agents and mechanical removal with the same rigour as biocides. Who it is for and how to use it The intended reader is a practitioner or a serious newcomer: a doctoral student inheriting a flow-cell rig, a clinical microbiologist asked to evaluate an antibiofilm dressing, an engineer in a water utility or a food plant responsible for a biocide programme, a reviewer who wants to know which questions to ask of a manuscript. The text assumes familiarity with basic microbiology and aseptic technique but not with fluid dynamics or image processing. Where numbers are given, they are either taken from named standards and published protocols or worked out in the text so that readers can repeat the arithmetic with their own parameters. It is not a cookbook in the sense of prescribing one correct protocol. Biofilm research serves too many purposes for that. A laboratory studying the genetics of matrix production on glass needs different compromises from a regulatory laboratory testing a hospital disinfectant, and both differ from a group modelling biofouling in a reverse osmosis plant. What the chapters offer instead is a way of reasoning about each choice: what it controls, what it confounds, how to check it, and what must be written down. The recurring advice is simple. Decide what quantity you are trying to measure before you choose the method. Run the controls that would reveal an artefact. Treat independent experiments, not image stacks or wells, as the unit of replication. Report the physical and chemical conditions completely enough that someone else could build your system from your methods section. Three commitments shape the text. First, precision about terms. "Biomass", "viability", "eradication" and "disruption" are used loosely in the literature, and much confusion flows from that looseness; each is defined operationally where it first matters. Second, honesty about limits. Several widely used methods have known biases that are rarely acknowledged in papers that use them, and those biases are described plainly, with the published evidence. Third, respect for the target surface. Every protocol here is ultimately a model of some real place, whether an airway, a catheter lumen, a heat exchanger or a ship's hull, and a model is only as useful as the argument connecting it to that place. Biofilm science has matured enormously since the first flow chambers were bolted to microscope stages. Open-source analysis software, commercial microfluidic plates, genetically encoded reporters and consensus standards now allow measurements that would have been heroic twenty years ago. The same maturity raises the bar. Journals, regulators and industrial clients increasingly expect biofilm data to come with the kind of methodological transparency long demanded in analytical chemistry. The chapters that follow are written to help meet that expectation, one design decision at a time. Chapter 1. What a Biofilm Measurement Measures Every biofilm protocol begins with a decision that is often made without being noticed: what, exactly, is to be quantified. "Biofilm formation" can mean the number of cells attached to a surface after two hours, the dry mass of material on a coupon after a week, the volume of green fluorescent protein signal in a confocal stack, the optical density of crystal violet eluted from a well, or the number of colony-forming units recovered after sonication. These quantities are correlated only loosely, and sometimes not at all. A mutant that attaches poorly can still build a thick biofilm given time; a treatment that strips matrix can lower crystal violet staining while leaving viable counts untouched; an antimicrobial that kills cells without removing them can leave biovolume unchanged while abolishing regrowth. The first job of a well-designed experiment is to name the quantity of interest and choose a model and endpoint that actually report it. Endpoints and what they respond to It helps to sort biofilm endpoints by the property of the community they detect. Structural endpoints measure how much material is present and how it is arranged: biovolume, thickness, surface coverage, roughness, the size of cell clusters. They come from microscopy, optical coherence tomography or, crudely, from the dry weight of scraped material. Compositional endpoints measure what the material is made of: total protein, carbohydrate, extracellular DNA, specific polysaccharides such as Psl, Pel or alginate in Pseudomonas aeruginosa, or poly-N-acetylglucosamine in staphylococci. Physiological endpoints report on activity: respiration measured by the reduction of resazurin or tetrazolium salts, ATP content, expression of fluorescent reporters, incorporation of labelled substrates. Viability endpoints, the most contested category, attempt to distinguish living from dead cells, either by culture (colony counts), by membrane integrity (propidium iodide exclusion), by metabolic activity, or by molecular methods such as propidium monoazide combined with quantitative PCR. Crystal violet staining deserves a special note because it is the most common biofilm assay in the literature and the most frequently over-interpreted. The dye binds negatively charged molecules, including cell surface components and matrix polymers, whether the cells are alive or dead. Christensen and colleagues introduced the microtiter plate format in 1985 to study staphylococcal adherence to plastic, and George O'Toole's widely used protocol popularized a simple sequence of growth in a 96-well plate, rinsing, staining with 0.1 percent crystal violet, rinsing again, solubilizing the bound dye in 30 percent acetic acid and reading absorbance near 550 to 595 nanometres. That assay is excellent for screening mutant libraries for gross defects in attachment and accumulation. It is poor at measuring killing, because dead biomass stains as well as live, and it is vulnerable to the rinsing step, which can remove loosely attached material inconsistently between wells and operators. Elke Peeters, Hans Nelis and Tom Coenye compared crystal violet, SYTO 9, fluorescein diacetate, resazurin, XTT and dimethyl methylene blue staining on microtiter-grown biofilms in 2008 and found, unsurprisingly, that each method reports a different aspect of the biofilm and that repeatability varied considerably between methods and organisms. The practical rule is to pair a total-biomass stain with a viability or activity measure whenever treatment effects are claimed. Colony counting after removal is the reference method for viability in regulatory testing and the natural language of disinfection science, where efficacy is expressed as a log10 reduction. Its weakness is the removal step. Biofilm cells must be detached from the surface and dispersed into single cells or small clumps before plating, usually by scraping, vortexing, sonication or a combination. Incomplete removal underestimates density; incomplete disaggregation counts a clump of hundreds of cells as one colony-forming unit; excessive sonication kills fragile cells. Standard methods therefore specify removal procedures in detail, and any in-house protocol should validate its own by checking the surface after removal, for example by staining the sonicated coupon, and by comparing counts across removal intensities. Image-based endpoints avoid removal entirely and preserve spatial information, which is why confocal laser scanning microscopy became the workhorse of mechanistic biofilm research. Their weaknesses lie elsewhere: in the choice of fluorophore, in optical attenuation through thick samples, in thresholding, and in sampling. A confocal stack typically covers a field a few hundred micrometres across, which is a vanishingly small fraction of a flow-cell channel. Chapters 5 to 7 are largely about controlling these weaknesses. The main laboratory models Biofilm model systems differ along a few axes that matter more than their brand names: whether medium flows past the biofilm or sits still; the magnitude and character of the shear force; whether the surface is removable for destructive analysis; whether the biofilm can be observed non-destructively in real time; how many independent conditions can be run at once; and whether a consensus standard describes the method. Table 1 summarizes the most widely used systems along these axes. Table 1. Common laboratory biofilm models and their main properties. Model Flow and shear Throughput Main readouts Standard 96-well microtiter plate Static or orbital shaking High Crystal violet, resazurin, CFU None universal; MIABiE guideline (Allkja 2020) Peg lid (Calgary Biofilm Device, MBEC) Rocking or shaking, low High CFU, regrowth, MBEC ASTM E2799 CDC Biofilm Reactor Baffle rotation, moderate to high, continuous flow Moderate (24 coupons) CFU per coupon, log reduction ASTM E2562, E3161, E2871 Drip flow reactor Low, air-liquid interface Low to moderate CFU, microscopy of coupons ASTM E2647 Rotating disk reactor Rotation, moderate to high Moderate CFU, coupon imaging ASTM E2196 Glass flow cell Laminar, low, defined Low Confocal time series Published protocols (Tolker-Nielsen and Sternberg) Microfluidic channels Laminar, tunable over wide range Moderate to high Live imaging, gradients None; device-specific Static microtiter models are cheap, fast and scalable, and they are the natural first screen. Their limitations are well known: nutrients deplete and waste accumulates, planktonic cells sediment onto the bottom of the well and are hard to distinguish from genuinely attached cells, and much of the measured "biofilm" in shaken plates forms as a ring at the air-liquid interface on the well wall rather than on the bottom. Peg-lid devices, developed by Howard Ceri and colleagues at the University of Calgary and published in 1999 as the Calgary Biofilm Device, avoid sedimentation by growing biofilm on 96 pegs projecting down into the wells of a trough or plate. Because the lid can be moved from plate to plate, the pegs can be rinsed, exposed to a dilution series of antimicrobials, neutralized and then sonicated into recovery medium, giving a convenient route to the minimum biofilm eradication concentration discussed in Chapter 8. Joe Harrison and colleagues published a detailed protocol for the device in Nature Protocols in 2010. The CDC Biofilm Reactor, developed at the Centers for Disease Control and Prevention and refined at the Center for Biofilm Engineering at Montana State University, is a one-litre vessel containing eight polypropylene rods, each holding three removable coupons, arranged around a baffled stir bar. Biofilm grows first in batch mode and then under continuous flow of dilute medium, and the coupons, available in materials from polycarbonate to stainless steel and titanium, are removed for sonication and plating or for microscopy. Darla Goeres and colleagues published a statistical assessment of its repeatability and ruggedness in 2005, and the method now underpins several ASTM standards and the EPA's guidance for biofilm claims. The drip flow reactor grows biofilm at an air-liquid interface under low shear on coupons tilted in channels, which suits models of wounds, lungs and industrial surfaces wetted by thin films. The rotating disk reactor offers higher and more uniform shear on coupons embedded in a spinning disk. Glass flow cells, the subject of Chapter 3, were the instrument behind much of the mechanistic understanding of P. aeruginosa biofilm development, including the classic mushroom-shaped structures formed in glucose medium and the roles of type IV pili and flagella in structuring them. Their virtue is continuous, non-destructive observation under defined laminar flow. Microfluidic channels extend this with smaller dimensions, lower volumes, precise control of shear and chemical gradients, and in commercial formats such as the BioFlux system many channels in parallel. The trade-offs, discussed in Chapters 2 and 3, include the gas and small-molecule permeability of the silicone polymer polydimethylsiloxane, the tendency of biofilm to clog narrow channels, and surface-to-volume ratios unlike most real environments. Heterogeneity, replication and the unit of analysis Biofilms are heterogeneous at every scale. Within a single stack, cell density varies from dense clusters to bare substratum. Between positions in the same channel, biofilm may be thicker near the inlet, where nutrients are richest, or near the outlet, where detached cells recolonize. Between channels, small differences in flow or inoculum can produce large differences in structure. Between experiments run on different days, variation in inoculum physiology, medium batches and temperature adds another layer. Arne Heydorn and colleagues demonstrated this layered variation formally for flow-chamber biofilms in 2000, in the same issue of Microbiology in which they introduced the COMSTAT program, and showed that variation between independent experiments could be substantial relative to variation within them. The consequence for design is that the unit of replication must be the independent experiment, meaning a separately prepared inoculum grown on a different occasion, not the image stack, the well or the coupon. Ten stacks from one channel in one experiment are ten measurements of one biofilm, useful for estimating that biofilm's mean but not for inferring anything about the process that generated it. A defensible design for a flow-cell comparison of two strains might use three independent experiments, two channels per strain per experiment, and five or more randomly positioned stacks per channel, analysed with a nested model or by averaging within each experiment before comparing strains. Microtiter work follows the same logic: technical replicate wells inform the precision of a single run, and biological replicates carried out on different days inform the conclusion. Colony counts should be log-transformed before averaging, because bacterial densities are approximately log-normally distributed and disinfection effects are multiplicative. Inoculum preparation is an underappreciated source of variation. Kasper Kragh and colleagues showed in 2018 that inoculating with pre-formed aggregates rather than single cells can change the outcome of biofilm experiments, including tolerance to antibiotics. Growth phase, washing, the optical density used and whether cells are vortexed all belong in the methods. The same is true of medium composition: many biofilm phenotypes depend strongly on carbon source, iron availability and divalent cations, and "LB" is not a single defined medium across suppliers and preparation methods. Time and the in vivo benchmark Time is a variable that many protocols fix by convenience rather than by reasoning. A 24-hour microtiter biofilm and a 7-day flow-cell biofilm of the same strain are different objects: the older one has had time to develop steep internal gradients, accumulate matrix, lose cells to dispersal and select for variants. Small-colony variants and wrinkly-spreader phenotypes of P. aeruginosa, for example, arise routinely in flow cells after several days and can come to dominate. A treatment tested only on young biofilm may therefore look far more effective than it would against a mature one, and a mutant phenotype visible at 24 hours may vanish by day four as compensatory processes catch up. Where possible, sample more than one time point, and state the age of the biofilm at treatment as carefully as the concentration of the treatment. A second benchmark is the biofilm found in real infections. Thomas Bjarnsholt and colleagues reviewed microscopy of samples from chronic wounds, cystic fibrosis sputum, otitis media and implant infections in 2013 and argued that in vivo biofilms are usually small aggregates, typically tens to a couple of hundred micrometres across, often not attached to any surface and embedded in host material such as mucus, necrotic tissue or neutrophil debris. That picture differs sharply from the continuous, surface-attached lawns grown in most laboratory reactors. It does not make reactor biofilms useless, but it means that claims about clinical relevance need to specify which feature of the in vivo state the model captures. An aggregate suspended in synthetic sputum and a mushroom-studded lawn in a glass flow cell will respond differently to an antibiotic whose penetration depends on diffusion distance. The same caution applies to industrial systems. Biofilm in a drinking water distribution pipe grows for months or years at low nutrient concentrations, carries a large inorganic load of corrosion products and sediment, and hosts a diverse community including protozoa that graze it. A two-day single-species biofilm grown in rich medium is at best a proxy for one aspect of that system, usually the response of a named pathogen to a named disinfectant under controlled conditions. Being explicit about the gap is not a weakness in a paper or a report; it is the information a reader needs to judge how far the result travels. Standards and reporting Two kinds of documents help discipline this complexity. The first are consensus standards, chiefly from ASTM Committee E35 on pesticides, antimicrobials and alternative control agents, which describe how to grow a named organism in a named reactor under stated conditions and how to test an antimicrobial against it. ASTM E2562 covers growth of P. aeruginosa in the CDC Biofilm Reactor; E3161 extends that practice to Staphylococcus aureus; E2871 describes the single tube method for disinfectant efficacy; E2647 the drip flow reactor; E2196 the rotating disk reactor; and E2799 the MBEC assay. Standards are sometimes dismissed in academic laboratories as regulatory paperwork, but they embody hard-won knowledge about ruggedness: which parameters matter, which can vary without consequence, and what repeatability to expect. Even a laboratory with no regulatory ambitions benefits from running a standard method alongside its own, because it anchors in-house results to a system others can reproduce. The second kind are reporting guidelines. The Minimum Information About a Biofilm Experiment initiative, published by Anália Lourenço and colleagues in 2014, proposed a checklist of what a biofilm paper should report: organism and strain source, medium, inoculum, surface, reactor geometry, flow and shear conditions, time, temperature, and the details of each analytical method. In 2020 Jontana Allkja and colleagues, working with the same community, published a minimum information guideline specifically for spectrophotometric and fluorometric microplate methods. These documents are short and unglamorous. Their value lies in forcing an author to notice what has been left out. A methods section that omits the flow rate, the channel dimensions or the concentration of stain is not merely incomplete; it describes an experiment nobody can repeat. Choosing a model deliberately The right model follows from the question. If the aim is to screen a transposon library for mutants defective in surface colonization, a static microtiter assay with crystal violet is appropriate, followed by confirmation of hits in a flow system. If the aim is to understand how matrix components shape three-dimensional architecture, a flow cell with fluorescently tagged strains and confocal imaging is the natural choice. If the aim is to support a label claim for a hospital surface disinfectant in the United States, the CDC Biofilm Reactor and single tube method specified by EPA guidance are effectively obligatory. If the aim is to predict biofouling in a membrane filtration system, a device that reproduces the spacer geometry, cross-flow velocity and feed water chemistry of the plant will be more informative than any generic reactor. A useful discipline is to write down, before the first inoculation, three things: the real environment the experiment is meant to inform, the feature of that environment the model is intended to reproduce, and the feature it deliberately omits. A flow cell perfused with minimal medium at room temperature reproduces laminar flow and continuous nutrient supply; it omits host cells, immune effectors, polymicrobial competition and the chemistry of body fluids. That is acceptable if the question concerns the genetic control of matrix production and unacceptable if the question is whether a dressing will clear a chronic wound. Stating the omission in advance keeps the interpretation honest. The same discipline applies to the endpoint. If a treatment is claimed to "disrupt" biofilm, the endpoint must detect removal of material, not only killing. If it is claimed to "kill", the endpoint must detect loss of the ability to regrow, not only a change in staining. If it is claimed to "prevent" biofilm, the treatment must be present during attachment and the comparison must be to untreated surfaces under identical flow. These distinctions sound pedantic, but most disputes about antibiofilm agents in the literature trace back to them. The rest of this booklet assumes the reader has made them and turns to the practical matter of getting the physical system right, beginning with the fluid that flows over the biofilm. Chapter 2. Shear, Flow and Transport: The Physics a Biologist Needs A flow rate is often the least-considered number in a biofilm methods section. It is copied from a previous thesis, chosen because the pump happened to have a convenient setting, or adjusted until the channel stopped clogging. Yet the flow rate, together with the geometry of the channel, sets the wall shear stress the attached cells experience, the rate at which nutrients and oxygen reach them, the time a planktonic cell spends in the system before being washed out, and the concentration at which a dosed antimicrobial actually arrives at the biofilm surface. Each of these shapes the community. This chapter provides the minimum physics required to choose and report flow conditions deliberately, with calculations a reader can repeat for any rectangular channel or pipe. Laminar flow and the Reynolds number Almost all laboratory flow cells and microfluidic devices operate in the laminar regime, where fluid moves in smooth layers without turbulent mixing. The Reynolds number, Re, is the ratio of inertial to viscous forces and indicates which regime applies. For a channel it is written Re = ρvD/μ, where ρ is the fluid density (about 1,000 kilograms per cubic metre for aqueous media), v the mean velocity, D a characteristic length and μ the dynamic viscosity. For a rectangular channel the characteristic length is the hydraulic diameter, D = 2wh/(w + h), where w is the width and h the height. Flow in pipes and ducts is reliably laminar below a Reynolds number of about 2,000. Take the three-channel flow cell described by Tim Tolker-Nielsen and Claus Sternberg, whose channels measure 1 millimetre high, 4 millimetres wide and 40 millimetres long. Their protocol uses a mean flow velocity of about 0.2 millimetres per second. The hydraulic diameter is 2 × 4 × 1 / 5 = 1.6 millimetres. With v = 0.0002 metres per second and μ = 0.001 pascal-seconds, Re = 1,000 × 0.0002 × 0.0016 / 0.001 = 0.32. The flow is deeply laminar; viscous forces dominate by three orders of magnitude over inertia. The same is true a fortiori in microfluidic channels tens of micrometres high. Real industrial systems are different. Water flowing at 1 metre per second through a 100-millimetre pipe has Re = 1,000 × 1 × 0.1 / 0.001 = 100,000, far into the turbulent regime. Turbulent flow produces fluctuating forces, thin viscous sublayers near the wall and efficient mixing, and biofilms grown under it are typically thinner, denser and more cohesive, often with streamlined filaments oscillating in the flow. Paul Stoodley and colleagues showed in the late 1990s, using mixed-species biofilms in glass flow cells run under laminar and turbulent conditions, that hydrodynamic regime and nutrient concentration together shape biofilm architecture, with laminar-grown biofilms forming patchy, roughly isotropic clusters and turbulent-grown biofilms forming elongated streamers and ripples. A biologist studying pipe biofilms therefore cannot assume that a laminar flow cell reproduces the physical environment, however carefully the wall shear stress is matched. Calculating wall shear stress The wall shear stress, τ, is the tangential force per unit area that the moving fluid exerts on the surface. It is measured in pascals (newtons per square metre); older literature uses dynes per square centimetre, and 1 pascal equals 10 dyn per square centimetre. For fully developed laminar flow between parallel plates, a good approximation for a rectangular channel much wider than it is high, the wall shear stress is τ = 6μQ / (w h²), where Q is the volumetric flow rate. The corresponding wall shear rate, γ, is 6Q / (w h²), in reciprocal seconds, and τ = μγ. Return to the Tolker-Nielsen flow cell. A mean velocity of 0.2 millimetres per second across a cross-section of 4 square millimetres gives Q = 0.8 cubic millimetres per second, which is 2.9 millilitres per hour, the "approximately 3 millilitres per hour" familiar from protocols. In SI units Q = 0.8 × 10^-9 cubic metres per second, w = 4 × 10^-3 metres and h = 1 × 10^-3 metres. Then τ = 6 × 0.001 × 0.8 × 10^-9 / (4 × 10^-3 × 1 × 10^-6) = 1.2 × 10^-3 pascals, or about 0.012 dyn per square centimetre. That is a very gentle shear, adequate for delivering nutrients and removing planktonic cells but far below the forces in many natural and engineered systems. Now consider a microfluidic channel 500 micrometres wide and 50 micrometres high perfused at 1 microlitre per minute, a common setting. Q = 1.67 × 10^-11 cubic metres per second. τ = 6 × 0.001 × 1.67 × 10^-11 / (5 × 10^-4 × 2.5 × 10^-9) = 0.08 pascals, nearly seventy times the Tolker-Nielsen value at a flow rate about three times lower, because shear scales with the inverse square of channel height. Halving the channel height at constant flow quadruples the shear. This sensitivity is the reason microfluidic devices can span a wide range of shear in a small footprint, and also the reason their shear values are easy to misstate when channel dimensions are quoted loosely. Two corrections are worth knowing. First, the parallel-plate formula assumes the width greatly exceeds the height. For an aspect ratio of 4, as in the Tolker-Nielsen channel, the formula gives a reasonable estimate along the centre of the floor but overstates shear near the side walls, where the velocity profile flattens into the corners. For aspect ratios near 1 an exact series solution or a numerical simulation should be used, and several microfluidic papers publish computational fluid dynamics maps of their channels for this reason. Second, viscosity depends on temperature and composition. Water at 37 °C has a viscosity of about 0.69 millipascal-seconds, roughly 30 percent lower than at 20 °C, and media containing mucin, serum or polymers can be considerably more viscous. For precise work, measure the viscosity of the medium at the operating temperature and use that value. For a pipe of diameter d under laminar flow, the corresponding relation is τ = 32μQ / (πd³), or equivalently τ = 8μv/d. Silicone tubing of 1-millimetre internal diameter carrying 3 millilitres per hour therefore experiences a wall shear of roughly 0.0085 pascals, higher than in the flow cell itself, which is one reason tubing upstream of a flow cell colonizes readily. For turbulent pipe flow, the wall shear is estimated from the Darcy friction factor f as τ = (f/8)ρv². For the 100-millimetre water main above, a friction factor of about 0.018, typical of smooth pipe at that Reynolds number, gives τ ≈ 0.018 / 8 × 1,000 × 1 = 2.3 pascals, three orders of magnitude above the laboratory flow cell. Biological environments span a similar range. Wall shear stress in large human arteries is usually quoted in the range of roughly 1 to 7 pascals and in veins well below 1 pascal. Urinary catheters experience intermittent low flow, the oral cavity experiences brief high forces during swallowing and brushing against long periods of near stasis, and chronic wounds are essentially static, wetted by exudate rather than swept by it. Henk Busscher and Henny van der Mei's 2006 review of microbial adhesion in flow displacement systems remains a useful summary of how shear is controlled and reported in adhesion studies and why the dependence of adhesion on flow is not monotonic. Moderate flow increases the delivery of cells to a surface while higher shear increasingly detaches them, so the number of attached cells can peak at an intermediate shear. Shear as a biological signal Shear stress is not only a mechanical nuisance that removes cells; bacteria sense it. The FimH adhesin of type 1 fimbriae in Escherichia coli forms catch bonds that strengthen under tensile force, so that adhesion to mannosylated surfaces is enhanced by moderate shear rather than weakened, as Wendy Thomas and colleagues showed in 2002. Surface sensing through type IV pili and flagella in P. aeruginosa feeds into cyclic di-GMP signalling and matrix production. Shear influences the expression of matrix genes, the density and cohesiveness of the matrix, and the formation of streamers. Streamers are a vivid example of flow shaping biofilm. Roberto Rusconi, Howard Stone and colleagues showed in 2010 that in curved microchannels, secondary flows near corners cause thread-like biofilm streamers to form and extend into the bulk flow, suspended between surfaces. In 2013 Knut Drescher and colleagues demonstrated that such streamers, formed by P. aeruginosa and S. aureus in channels designed to mimic porous media and medical devices, can clog a channel rapidly and catastrophically, well before surface-attached biofilm would have occluded it. Their results carry a methodological warning: in any device with bends, pillars, junctions or spacers, the most consequential biofilm may be the part that is not on the floor where the microscope is focused. The practical implication is that shear should be chosen to match the question and reported as a number, not as a pump setting. If the aim is to model a catheter, the shear and flow regime of urine in a catheter lumen, including periods of stasis, should be estimated and reproduced. If the aim is to screen matrix mutants, a standard, reproducible low shear such as that of the Tolker-Nielsen protocol is appropriate because comparability across studies is valuable. If the aim is to test whether a coating resists fouling in a heat exchanger, the laboratory device must reach shear stresses in the pascal range, which flow cells of millimetre height at millilitre-per-hour flows cannot do. Shear in stirred, rotating and shaken systems Not every device has a geometry simple enough for a closed-form shear calculation. In the CDC Biofilm Reactor the baffle rotates at a set speed, 125 revolutions per minute in the standard P. aeruginosa method, and the coupons, held vertically in rods around it, experience a flow that is turbulent in parts of the vessel and varies with coupon position and depth. The standard does not claim a single shear stress; it controls shear indirectly by fixing the vessel, baffle, rod geometry and rotation speed. That is a sound approach for a standardized method, because reproducibility comes from identical hardware, but it means shear values quoted for the reactor in papers are estimates from modelling and should be labelled as such. The rotating disk reactor gives a better-defined field: for a disk spinning in a large volume, shear stress at the surface rises with radius and with the rotation speed raised to about the power one and a half, so coupons at different radial positions experience different forces, a feature some studies exploit to test several shear levels at once. Orbital shaking of multiwell plates, widely used to "add flow" to static assays, produces an even less uniform field. The liquid sloshes as a travelling wave, so the floor of a well experiences oscillating shear whose magnitude differs between the centre and the edge and depends on fill volume, well diameter, orbit radius and speed. Computational studies of shaken six-well plates used in endothelial cell research have shown shear varying several-fold across a single well, with the centre relatively quiescent and the periphery swept more strongly. Biofilms in shaken plates frequently reflect this pattern, forming rings or gradients that a crystal violet reading averages away. When a shaken format is used, report fill volume, orbit diameter and speed, and consider imaging the whole well floor at low magnification to check whether the biofilm distribution is uniform before trusting a single absorbance value. The broader lesson is that "dynamic" is not a single condition. A shaken plate, a stirred reactor and a laminar flow cell all expose biofilms to moving fluid, but the magnitude, direction, steadiness and spatial uniformity of the resulting forces differ by orders of magnitude. Comparisons across systems should be made on the basis of estimated wall shear stress and transport conditions, not on the label attached to the device. Residence time, washout and nutrient delivery The mean residence time of liquid in a channel is its volume divided by the flow rate. The Tolker-Nielsen channel holds 1 × 4 × 40 = 160 cubic millimetres, or 160 microlitres. At 2.9 millilitres per hour, the residence time is about 3.3 minutes. No planktonic bacterium doubles in 3 minutes, so cells detached from the biofilm or growing in suspension are washed out long before they can build a planktonic population. This is intended: it means that what accumulates in the channel is surface-associated growth, and that the effluent carries recently detached cells whose numbers can be used as a measure of dispersal. Stirred reactors behave differently. The CDC Biofilm Reactor, with an operating volume of about 350 millilitres and a continuous-flow rate of about 11.7 millilitres per minute as specified for P. aeruginosa in ASTM E2562, has a residence time of about 30 minutes. The dilution rate, flow divided by volume, is then about 2 per hour, which exceeds the maximum growth rate of most bacteria in the dilute medium used, so planktonic cells are washed out and the steady-state community is dominated by the biofilm on the coupons and the reactor walls. If a laboratory lowers the flow rate to save medium, the dilution rate can fall below the growth rate, a dense planktonic culture establishes itself, and the biofilm grows under different nutrient and oxygen conditions. The residence time is therefore not a detail; it determines whether the reactor is a biofilm reactor at all. Nutrient delivery depends on flow in a subtler way. Near any surface lies a concentration boundary layer through which nutrients reach the biofilm by diffusion. Faster flow thins that layer and increases the flux of substrate to the surface; slower flow thickens it. The relative importance of flow-driven transport and diffusion is captured by the Péclet number, Pe = vL/D, where D here is the molecular diffusion coefficient of the solute. For glucose, with a diffusion coefficient in water of about 6.7 × 10^-10 square metres per second, in a flow cell at 0.2 millimetres per second over a characteristic length of 1 millimetre, Pe is about 300. Advection dominates along the channel, but within the biofilm itself, where there is little or no flow, transport is by diffusion alone, and the ratio of reaction to diffusion determines how deeply a substrate penetrates. Philip Stewart's 2003 review of diffusion in biofilms explains why this matters. Oxygen is consumed rapidly by respiring cells and has limited solubility, so it often penetrates only tens of micrometres into an actively respiring biofilm. Below that depth cells are anoxic or starved, grow slowly, and are phenotypically tolerant to antibiotics that target growth. Microelectrode measurements and reporter-gene studies of P. aeruginosa biofilms have shown stratified growth activity concentrated in the oxygenated outer layer. When a protocol changes flow rate, it changes the thickness of the oxygenated zone and hence the proportion of the biofilm that is physiologically active. That in turn changes susceptibility to antimicrobials, which is one reason why tolerance results from static and flow systems differ. Constant flow, constant pressure and pulsation Pumps impose either a flow rate or a pressure, and the choice matters once the biofilm starts to obstruct the channel. Peristaltic and syringe pumps are flow-controlled: they push a set volume per unit time regardless of resistance. As biofilm narrows a channel, the pressure required to maintain that flow rises, the velocity through the remaining gap increases, and the local shear stress climbs. For a rectangular channel under laminar flow, the pressure drop is ΔP = 12μLQ / (w h³). In the 500-by-50-micrometre microchannel above with a length of 20 millimetres, 1 microlitre per minute requires only about 64 pascals. If biofilm halves the effective height, the pressure drop at the same flow rises eightfold and the shear on the biofilm surface fourfold. Pressure-controlled systems, by contrast, keep the driving pressure fixed, so flow and shear fall as the channel occludes, and the biofilm can grow into a nearly stagnant space. Neither is wrong, but they model different real situations: a pumped industrial circuit resembles constant flow; a gravity-fed drainage line or a porous aquifer is closer to constant pressure. Peristaltic pumps, the workhorses of flow-cell laboratories, deliver flow in pulses as rollers compress the tubing. At low speeds with small tubing the pulsation is modest, but it can be visible in live imaging as rhythmic movement of loosely attached cells. A length of compliant tubing or a bubble trap between pump and flow cell damps pulsation. Tubing wears and its internal diameter changes over days of use, so flow rates should be checked gravimetrically at the start and end of long experiments by weighing the effluent collected over a timed interval. A single number in a methods section, "3 millilitres per hour, verified by collection of effluent at 0 and 72 hours", tells a reader far more than a pump model and a speed setting. The discipline recommended here is modest. For every flow experiment, record the channel dimensions, the verified flow rate, the medium viscosity at the operating temperature, and the resulting calculated wall shear stress, Reynolds number and residence time. Report them. When comparing to real environments, estimate the corresponding values there, and state honestly where the laboratory system departs from them. With those numbers in hand, the practical work of building a flow system, the subject of the next chapter, becomes a matter of engineering to a specification rather than following a recipe. Chapter 3. Building and Running Flow Cells and Microfluidic Systems Flow-cell work has a reputation for being temperamental, and the reputation is deserved when systems are assembled casually. Most failures fall into a short list: air bubbles that tear biofilm off the substratum, contamination that enters through a leaking joint, bacteria growing backwards up the tubing into the medium reservoir, channels that clog at the inlet, and flow rates that drift as pump tubing wears. None of these is mysterious, and nearly all can be prevented by design. This chapter describes the classic glass-and-polycarbonate flow-cell system in enough detail to build and run one, then considers microfluidic devices, and closes with a troubleshooting discipline that turns a temperamental method into a routine one. Anatomy of a flow-cell system The system popularized by Søren Molin's group in Copenhagen and described in protocols by Claus Sternberg and Tim Tolker-Nielsen consists of five elements connected in series: a medium reservoir, a multichannel peristaltic pump, bubble traps, the flow cells themselves, and an effluent container. Medium is drawn from the reservoir through silicone tubing, pushed by the pump through a bubble trap, and delivered to the inlet of each flow channel; it leaves through the outlet and runs to waste. Each channel is independent, so a three-channel flow cell with three bubble traps and three pump lines can carry three strains or three conditions side by side. The flow cell is a small block, commonly machined from polycarbonate, with channels cut through it and a glass coverslip glued over the open face to form the floor on which the biofilm grows and through which it is imaged. A standard 24-by-50-millimetre coverslip is attached with silicone adhesive, and the glue must cure fully, typically overnight, before the cell is used. Coverslip thickness matters for the microscope, as Chapter 6 explains; most high-numerical-aperture objectives are corrected for glass 0.17 millimetres thick, the nominal thickness of a number 1.5 coverslip, and protocols using thinner number 1 glass should note it. Luer or barbed connectors at each end of the channel accept silicone tubing. Silicone tubing is used because it is flexible, autoclavable and gas-permeable, which helps oxygenate the medium on its way to the channel. Its gas permeability cuts both ways: it also allows water vapour to escape and carbon dioxide to exchange, and bubbles can form within it if the medium is warmer downstream than upstream. Pump segments, the short lengths of tubing compressed by the pump rollers, are usually a harder-wearing material and of a specified internal diameter, because the flow delivered per revolution depends directly on that diameter. Bubble traps sit immediately upstream of the flow cells. The classic design is a vertical chamber, often a modified syringe barrel, into which medium enters at the top and leaves at the bottom, so that any air bubble rises and is retained rather than passing into the channel. A bubble reaching the biofilm acts like a squeegee: its air-liquid interface carries enough surface tension to strip cells from the substratum along its path. Sarah Crusz and colleagues, reviewing the problem in 2012, described a flow-cell methodology designed specifically to eliminate bubbles, and their work illustrates how much of flow-cell success depends on this single failure mode. Assembly, sterilization and conditioning Flow cells are generally not autoclaved because polycarbonate and silicone adhesive tolerate heat poorly and the coverslip can crack or detach. Instead, the assembled system is sterilized chemically. The standard approach is to fill the entire circuit, reservoir line, pump tubing, bubble traps, flow cells and outlets, with a hypochlorite solution at around 0.5 percent available chlorine, leave it for several hours, then flush thoroughly with sterile water to remove residual chlorine before filling with medium. Tubing can be autoclaved separately before assembly, and connectors can be autoclaved or soaked. Some laboratories follow the hypochlorite step with a rinse of sterile water run overnight to ensure no oxidant remains, because residual chlorine will kill the inoculum or select for tolerant cells. Once filled with medium, the system should run for several hours, ideally overnight, before inoculation. This serves three purposes. It checks for leaks and bubbles while nothing is at stake. It confirms sterility: the effluent can be plated, and any turbidity in the channels signals contamination before the experiment begins. And it allows a conditioning film of medium components to adsorb onto the glass, as it would in any real system, so that the first cells to arrive meet a conditioned rather than a pristine surface. For experiments on biological coatings, discussed in Chapter 4, conditioning is deliberate and the coating is applied at this stage. The medium itself deserves attention. The FAB minimal medium used in many P. aeruginosa flow-cell studies, supplemented with a defined carbon source such as 0.3 millimolar glucose or citrate, gives reproducible and interpretable structures precisely because it is defined. Rich media such as LB or tryptic soy broth support faster growth and thicker biofilm but vary more between batches and produce denser planktonic populations in any part of the system with a long residence time. The concentration of carbon source matters: with glucose, P. aeruginosa forms the well-known mushroom structures with stalks and caps; with citrate, flatter biofilms form, as Mikkel Klausen and colleagues showed in 2003. Medium should be prepared in volumes sufficient for the whole experiment and warmed to the operating temperature before it reaches the flow cell, which reduces bubble formation from gas coming out of solution. Inoculation and the first hours Inoculation is the moment of greatest risk. The standard procedure is to stop the pump, clamp the tubing downstream of the flow cell, and inject a small volume of diluted culture, commonly a few hundred microlitres of a culture adjusted to an optical density at 600 nanometres around 0.01 to 0.001, through the tubing just upstream of the inlet using a fine needle and syringe. The injection site in the silicone tubing is sealed afterwards with a small dab of silicone glue. The flow cell is then commonly left inverted, coverslip uppermost, for about an hour without flow, so that cells settle onto and attach to the substratum rather than the polycarbonate floor; it is then turned coverslip-down again and the pump restarted. Several details decide whether inoculation is reproducible. The culture should be in a defined growth phase and washed if the aim is to avoid carrying spent medium into the channel. The inoculum density should be verified by plating, not only by optical density. The injection must avoid introducing air, which means expelling bubbles from the syringe and needle before insertion and injecting slowly. If aggregates are present in the culture, as Kragh and colleagues showed they often are, they should either be dispersed deliberately, by vortexing or brief sonication, or included deliberately and their presence reported, because initial aggregates alter subsequent development. After restarting flow, unattached cells are washed out within minutes; the effluent over the first hour can be plated to estimate the fraction of the inoculum that attached. Many laboratories include, within every experiment, one channel inoculated with a reference strain grown under the standard conditions of that laboratory. This internal reference allows day-to-day variation to be monitored: if the reference strain's biovolume on day three drifts well outside its historical range, something about the medium, temperature or inoculum has changed, and comparisons between test strains in that run should be treated with suspicion. Fluorescent tags and reporters Non-destructive imaging usually relies on bacteria that fluoresce constitutively, which in Gram-negative organisms is often achieved by inserting a gene for green fluorescent protein or a red fluorescent protein into a neutral chromosomal site using mini-Tn7 transposon vectors, such as those described by Birgit Koch and colleagues in 2001. Chromosomal insertion avoids the plasmid loss and copy-number variation that make plasmid-borne tags unreliable in long flow experiments without antibiotic selection. Two strains tagged with different colours can be co-inoculated to study competition or spatial organization. Two caveats accompany fluorescent proteins. First, GFP and most of its relatives need molecular oxygen for chromophore maturation. In the anoxic interior of a thick biofilm, newly synthesized protein may fail to fluoresce, so a dark core can mean anoxia rather than absence of cells. Flavin-based fluorescent proteins, which do not require oxygen, and self-labelling tags that bind fluorescent ligands offer alternatives where this matters. Second, stable fluorescent proteins persist for hours to days after a cell stops growing or dies, so constitutive fluorescence reports presence, not activity. Claus Sternberg and colleagues showed in 1999 that fusing a ribosomal RNA promoter to an unstable GFP variant, with a short-lived protease-sensitive tag, reports local growth activity and reveals the stratification of growth within flow-chamber biofilms. Reporter design is therefore part of the protocol, not a background detail. Microfluidic devices Microfluidic channels, typically tens to a few hundred micrometres in height, allow smaller volumes, higher and more precisely defined shear, rapid medium switching and the construction of gradients, mixers, pillars, constrictions and porous-medium mimics. Most academic devices are made by soft lithography: a master mould is patterned in photoresist on a silicon wafer, polydimethylsiloxane is cast against it and cured, and the moulded PDMS is bonded to a glass coverslip after oxygen plasma treatment. Inlet and outlet holes are punched and connected to tubing driven by syringe pumps or pressure controllers. PDMS brings three problems for biofilm work. It is highly permeable to oxygen, so a biofilm in a PDMS channel may be better oxygenated from the ceiling than a biofilm in a glass flow cell or a real pipe, which alters physiology and antibiotic tolerance. It absorbs small hydrophobic molecules: Michael Toepke and David Beebe showed in 2006 that hydrophobic compounds partition into PDMS, depleting them from the medium, which can make an antimicrobial or a signalling molecule appear less potent than it is. And it is permeable to water vapour, so evaporation through thin PDMS layers during long experiments can concentrate the medium and nucleate bubbles. Remedies include thermoplastic devices made from cyclic olefin copolymer or poly(methyl methacrylate), surface coatings that reduce absorption, keeping the device in a humidified chamber, and pre-saturating PDMS by soaking it in medium or in the test compound before use. Commercial systems reduce fabrication burden. The BioFlux platform, described by Michael Benoit and colleagues in 2010, integrates microfluidic channels into a standard well-plate footprint and drives flow pneumatically from inlet wells to outlet wells, allowing many channels to run under defined shear simultaneously on a microscope stage. Channel slides with integrated reservoirs and dedicated pump systems from several manufacturers serve similar ends. Hervé Straub and colleagues described in 2020 a microfluidic platform designed specifically for in situ investigation of biofilm formation and treatment, illustrating how a purpose-built device can combine controlled growth, dosing and imaging. Commercial devices standardize geometry, which aids reproducibility, but users should still verify the manufacturer's shear values for their own medium and temperature, and should check whether the channel material absorbs their test compounds. Clogging is the characteristic failure of microfluidic biofilm experiments, and it often begins at the inlet, where cells meet fresh medium first and where tubing connections create dead zones. Designs that keep the inoculum from contacting the inlet region, for instance by seeding from the outlet side or by using a sheath flow of sterile medium near the inlet, delay clogging. When the question concerns clogging itself, as in Drescher's streamer experiments, the pressure drop across the device becomes a primary measurement and should be logged continuously. Time-lapse imaging, in situ staining and harvest A flow cell earns its keep when the same biofilm is followed over time. Time-lapse imaging requires three things that are easy to neglect. The first is a stable environment on the stage: a heated enclosure or stage-top incubator, a pump that keeps running during acquisition, and tubing routed so that it does not tug the flow cell as the stage moves. The second is a way to return to the same positions. Motorized stages can store coordinates, but thermal expansion and the mechanical play of the flow-cell holder mean that a stored position may drift by tens of micrometres over a day; marking the coverslip with a fine pen outside the channel, or recording a low-magnification overview at each session, helps relocate fields. Hardware autofocus systems that track the coverslip surface are valuable for long series, because the lower boundary of the biofilm is the natural reference plane. The third is restraint with light. Each stack delivers a dose of excitation light that can bleach fluorophores and, at higher doses, damage cells; imaging every few hours with modest laser power usually yields more trustworthy growth curves than imaging every twenty minutes at full power. Stains can be applied without dismantling the system. For LIVE/DEAD or matrix staining at an end point, the pump is stopped, the tubing clamped downstream, and the staining solution injected upstream of the inlet in the same way as the inoculum, then left in the channel for the prescribed time before gentle flushing with medium or buffer. The volume injected should exceed the channel volume by a comfortable margin so that the stain concentration in the channel is known rather than diluted by residual medium. Because injection is itself a disturbance, a control channel that receives buffer alone by the same route shows whether the procedure dislodges biofilm. Harvest for destructive analysis is less standardized. Effluent can be collected continuously or at intervals to count detached cells, which provides a dispersal time course without touching the biofilm. At the end of an experiment, biomass can be removed for colony counts, nucleic acid extraction or biochemical assays by flushing the channel vigorously with buffer, sometimes with small glass beads introduced to scour the surface, or by prising off the coverslip and scraping it. Both methods remove an unknown fraction of the biofilm, so recovery should be checked by staining and imaging the channel after harvest. For studies that need both images and counts from the same biofilm, image first, then harvest, and state the order. A troubleshooting discipline Flow systems reward a few habits consistently applied. The following checks, built into every run, prevent most lost experiments: • Verify flow rates gravimetrically at the start and end of each experiment, and after any change of pump tubing. • Warm medium to the operating temperature and, for long runs, degas it before use; inspect bubble traps daily and empty them before they fill. • Plate effluent from every channel at inoculation, midway and at harvest; a contaminant colony type on the plate is information that can save a misinterpreted result. • Watch the upstream tubing for grow-back. Motile bacteria can swim and grow against the flow, colonizing the tubing between pump and reservoir; a physical break in the line, such as a drip chamber in which medium falls through air, prevents them reaching the reservoir. • Log temperature at the flow cell, not at the incubator thermostat. Stage-top incubators and room air conditioning create gradients that change growth rates. • Keep a written run sheet recording, for each channel, strain, inoculum density verified by plating, time of inoculation, any interruption of flow, and every intervention. A final point concerns interruptions. Stopping flow for even an hour, to move a flow cell to the microscope or change a reservoir, alters the environment: oxygen falls, nutrients deplete near the biofilm, and cells begin to respond. Where possible, image on a microscope equipped with a stage incubator and keep the pump running during acquisition. Where flow must stop, record when and for how long, and keep the interruption the same across compared channels. The biofilm has no way of knowing that a pause was for the experimenter's convenience; it responds to the conditions it experiences, and those conditions are the experiment. Hashtags: #BiofilmResearchProtocols #BiofilmResearch #MicrofluidicFlowCells #FlowCellSystems #BiofilmMicrofluidics #BiofilmHydrodynamics #WallShearStress #ReynoldsNumber #ResidenceTime #BiofilmFormation #ConfocalMicroscopy #ConfocalBiomass #BiofilmImaging #COMSTAT2 #BiofilmQ #LiveDeadStaining #BiofilmViability #BiofilmBiovolume #AntimicrobialTesting #BiofilmDisruption #BiofilmEradication #CalgaryBiofilmDevice #CDCBiofilmReactor #BiofilmReproducibility #FutureOfBiofilmResearch