Download the Book (PDF): Introduction Every protocol that involves a living animal begins with a question that is easy to skip and hard to answer well: why this animal, why this many, and why this way? The question is not a formality. It sits at the centre of the modern law of animal research in Europe, North America and much of the rest of the world, and it is the practical form of a principle first set out more than sixty years ago. Asked seriously, it improves the lives of the animals involved. Asked seriously, it also improves the science. That second point is the argument of this book. The framework known as the Three Rs, Replacement, Reduction and Refinement, is usually presented as an ethical constraint on research: a set of obligations that scientists owe to animals and that regulators enforce, sometimes against the grain of scientific convenience. That picture is not wrong, but it is incomplete in a way that matters. Properly understood, the Three Rs are a discipline of good experimental practice. An animal that suffers unnecessarily is a source of uncontrolled variation. A study that uses too few animals to detect the effect it is looking for wastes every animal in it. A model that does not resemble the human condition it is meant to represent produces results that will not translate, however many animals are used. A replacement method that captures the relevant human biology more faithfully than a rodent is not a compromise; it is a better instrument. Welfare and validity are not rival goods to be traded off against each other. In most of the decisions that matter, they point the same way. Where the idea came from The Three Rs were formulated by the zoologist William Russell and the microbiologist Rex Burch in The Principles of Humane Experimental Technique, published in 1959 after a study commissioned by the Universities Federation for Animal Welfare. Their book was neither a polemic against animal research nor a defence of it. It was an attempt to apply scientific thinking to the problem of inhumanity in the laboratory: to classify its sources, to identify the techniques that could reduce it, and to argue that the most humane science and the best science tended to coincide. Russell and Burch wrote that "the greatest scientific experiments have always been the most humane and the most aesthetically attractive", and they meant it as an empirical observation rather than a pious hope. For two decades the book was largely neglected. Its revival from the late 1970s onwards, and its absorption into law, guidance and institutional practice from the 1980s, turned three words into an organising principle for a whole sector. Today the Three Rs appear in the text of European Union Directive 2010/63/EU, which obliges member states to ensure that the principles of replacement, reduction and refinement are applied systematically. They are woven through the United States' regulatory system, from the Animal Welfare Act's requirement that investigators consider alternatives to painful procedures to the Guide for the Care and Use of Laboratory Animals, which endorses them explicitly. National centres now exist to advance them, most prominently the UK's National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs), founded in 2004. Journals ask authors to report their studies according to the ARRIVE guidelines, which exist in large part because poor reporting makes animal studies impossible to evaluate or reproduce. What this book covers The chapters that follow set out what a rigorous, welfare-centred approach to animal research looks like in practice, from the original concepts through the regulatory architecture to the concrete decisions made at the bench, in the animal facility and in the ethics committee. Chapter 1 returns to Russell and Burch to recover what the Three Rs actually meant and why their authors saw humaneness and scientific quality as connected. Several ideas in their book, including the distinction between high-fidelity and discrimination models and the warning against assuming that a more similar animal is always a better model, are more useful today than the slogan that survives them. Chapter 2 describes the regulatory frameworks within which animal research operates: the European Directive with its harm-benefit analysis and severity classification, the parallel but differently constructed systems of the United States, and the United Kingdom's licensing regime. The purpose is not to turn readers into compliance officers but to show how the law turns an ethical principle into a sequence of decisions, and where those decisions are made. Chapters 3 and 4 take up Replacement. The first surveys the landscape of established non-animal methods, with particular attention to regulatory toxicology, where validated replacements have displaced animal tests entirely for some endpoints. The second turns to the newer human-based technologies that are reshaping biomedical research: organoids grown from stem cells, microfluidic organ-on-chip systems, and computational models that simulate physiology and predict toxicity. It asks what these methods can already do, what they cannot yet do, and how their validity should be judged. Chapters 5 and 6 are about Reduction, which is best understood as the application of sound experimental design and statistics. Chapter 5 addresses bias, randomisation, blinding and transparent reporting, drawing on the evidence that much published animal research has been compromised by weaknesses that are cheap to prevent. Chapter 6 treats sample size in detail: why studies that are too small waste animals as surely as those that are too large, how power calculations work and what they require, and which design strategies allow a given question to be answered with fewer animals. Chapters 7 and 8 address Refinement. Chapter 7 concerns the recognition and prevention of pain, the planning of anaesthesia and analgesia as a matter of institutional policy and veterinary partnership, and the evidence that withholding pain relief is far less often scientifically justified than was once assumed. Chapter 8 deals with humane endpoints, severity assessment and the animal's whole lifetime experience, from breeding and housing to handling and the manner of its death. Throughout these chapters, pain management is discussed at the level of principle, planning and professional responsibility. The specific choice of drugs, doses and schedules is a clinical matter for the attending veterinarian working with the research team and the relevant formulary, and it is not the business of a book like this to substitute for that consultation. Chapter 9 turns to the institution: the ethics committees and welfare bodies that review protocols, the training and competence of the people who handle animals, the "culture of care" that determines whether the rules are followed in spirit as well as letter, and the role of openness, retrospective review and systematic evidence synthesis in making the whole enterprise accountable. The conclusion draws these threads together and considers what remains unsettled, including the fast-moving policy shifts toward human-based methods in regulatory science and the unresolved questions about how new approaches should be validated. Who it is for The book is written for anyone who designs, reviews, supports or oversees research involving animals: early-career scientists writing their first protocols, established investigators who want to see how the field's expectations have moved, members of ethics committees, including the lay and non-scientific members who bring an essential outside perspective, animal technologists and care staff, veterinarians new to laboratory animal medicine, research managers, and funders. It should also be useful to students and to members of the public who want to understand how animal research is regulated and how the people who conduct it think about their responsibilities. It assumes no specialist knowledge of statistics, pharmacology or law. Where technical ideas matter, such as statistical power or the validation of an alternative method, they are explained in plain terms. The regulatory detail is accurate to the published frameworks, but readers should always consult the current text of the rules that apply to them and the guidance of their own institution, because both change. A note on stance This is not a book that argues for or against the use of animals in science as such. Reasonable people disagree about that question, and the disagreement is not going to be settled by a better protocol. What can be said with confidence is that animals are currently used in large numbers in research, that the law in most jurisdictions permits this only under conditions, and that within those conditions there is almost always room to do better: to replace an animal study with a method that answers the question more directly, to design an experiment that needs fewer animals and gives a clearer answer, to spare an animal pain that serves no purpose. The people best placed to make those improvements are the people who do the work. This book is written for them, in the conviction that compassion and rigour are not separate virtues in the laboratory but the same habit of taking the details seriously. Chapter 1: What Russell and Burch Actually Argued The Three Rs are among the most widely cited ideas in the life sciences and among the least read. Many researchers who can recite the three words have never opened the book in which they were introduced, and much of what that book contains has been lost in the translation from argument to slogan. Recovering it is worth the effort, because the original is more demanding, more scientific and more practical than the version that circulates in training slides. The study and its setting In 1954 the Universities Federation for Animal Welfare, a British charity founded by Charles Hume with a distinctive commitment to working with scientists rather than against them, commissioned a systematic study of humane technique in laboratory experimentation. The task went to William Russell, a zoologist with broad interests in psychology and the classics, assisted by Rex Burch, a microbiologist. Their report, The Principles of Humane Experimental Technique, appeared in 1959. The period matters. Animal use in British and American laboratories was expanding rapidly after the Second World War, driven by the growth of pharmacology, the demands of vaccine production and biological standardisation, and a new scale of government-funded biomedical research. The regulatory framework in Britain was still the Cruelty to Animals Act of 1876, which licensed individual experimenters but said little about how experiments should be designed. The United States had no federal law governing laboratory animal welfare at all; the Animal Welfare Act would not arrive until 1966. Into this setting Russell and Burch brought something unusual: an attempt to treat inhumanity itself as a phenomenon that could be studied, classified and reduced by scientific means. The definitions Russell and Burch defined their three principles with some care, and the definitions are worth stating accurately. Replacement meant any scientific method that uses non-sentient material in place of conscious living vertebrates. They included under this heading the use of higher plants, microorganisms, tissue and cell cultures, and what they called "physico-chemical" techniques, and they distinguished between absolute replacement, in which no animal is used at any stage, and relative replacement, in which animals are still required, for instance as a source of tissue, but are not subjected to any procedure that could cause distress while alive. Reduction meant lowering the number of animals needed to obtain information of a given amount and precision. The qualification is essential and frequently forgotten. Reduction is not simply using fewer animals; it is using fewer animals for the same quality of answer. A study that halves its animal numbers but can no longer detect the effect it was designed to find has not achieved reduction. It has converted a useful experiment into a wasteful one. Refinement meant any decrease in the incidence or severity of inhumane procedures applied to those animals that still have to be used. Russell and Burch were explicit that refinement applies to everything that happens to the animal, not only to the experimental procedure, and that its aim is to reduce the total amount of distress the animal experiences. Two features of this scheme deserve emphasis. First, the definitions are ordered by a logic of priority. Replacement comes first because, where it is possible, it removes the problem entirely. Reduction and refinement apply to the animals that still have to be used. Second, the three principles are not independent. Reduction can conflict with refinement, for example when fewer animals are each subjected to more procedures, and later writers have spent considerable effort on how such conflicts should be resolved. Russell and Burch themselves were clear that the goal was to minimise the total inhumanity of the enterprise, not to maximise any one of the three Rs at the others' expense. Humane and inhumane The book's central category was not "the Three Rs" but inhumanity, which Russell and Burch analysed with some subtlety. They distinguished direct inhumanity, the distress inflicted as an unavoidable consequence of a procedure, from contingent inhumanity, which arises as an incidental and inessential by-product of the way the work is done: poor husbandry, rough handling, inadequate anaesthesia, avoidable infection, neglect of the animal's needs outside the experimental procedure itself. Much of the suffering in laboratories, they argued, was contingent. It served no scientific purpose, and in many cases it actively undermined the science by introducing stress and disease into the very animals whose physiology was being measured. This distinction remains one of the most practically useful ideas in the field. When a protocol is reviewed today, the question of direct harm, what the procedure itself does to the animal, is usually well considered, because it is written down and scrutinised. Contingent harm is harder to see. It lives in the details: the temperature of the animal room, the method of picking a mouse up, whether a post-operative animal can reach its food and water, whether the person checking the animals at the weekend knows what signs to look for. Refinement in the fullest sense is largely the systematic elimination of contingent inhumanity, and it is where the most improvement is usually available at the least scientific cost. Fidelity and discrimination One of the more sophisticated sections of The Principles of Humane Experimental Technique concerns the nature of models. Russell and Burch distinguished between two properties a model might have. A high-fidelity model resembles the system being modelled in many respects; the classic assumption is that an animal more closely related to humans, or more similar physiologically, is a better model. A discrimination model, by contrast, is one that reproduces the specific property under investigation, even if it differs from the target in almost every other respect. They argued that researchers were prone to what they called the "high-fidelity fallacy": the assumption that a model's overall similarity to humans guarantees its relevance to a particular question. In fact, what matters is whether the model behaves like the target system with respect to the mechanism being studied. A cell line expressing the human version of a receptor may be a far better model for a drug's action at that receptor than a whole rodent whose receptor differs in its binding properties. Conversely, an animal that shares many features of human physiology may diverge from humans on exactly the pathway that matters, and its apparent fidelity then becomes a source of false confidence. This argument is the intellectual foundation for replacement as a scientific strategy rather than a moral concession. It explains why, in some fields, human-based in vitro and computational methods have outperformed animal models on specific predictive tasks, and why the value of an animal model has to be established for each question rather than assumed. The failure of many treatments that worked in animal models of stroke, sepsis, amyotrophic lateral sclerosis and other conditions to translate into benefit in patients is, in part, a demonstration of the high-fidelity fallacy at scale. Researchers treated whole-animal models as faithful representations of human disease when the relevant mechanisms differed in ways that mattered. Why the book was neglected, and why it returned For most of the 1960s and 1970s, the Three Rs had little visible influence. Russell himself later suggested various reasons: the book was long and dense, the scientific community of the time was not receptive, and the anti-vivisection movement had little interest in a framework that accepted animal research as legitimate while seeking to improve it. The period's expansion of research also made reflection on its methods seem less urgent than getting on with the work. The revival came from several directions. In the late 1970s and 1980s, public controversy over animal testing, particularly of cosmetics and household products, created pressure for alternatives. Organisations such as the Fund for the Replacement of Animals in Medical Experiments in the UK, founded in 1969, and the Johns Hopkins Center for Alternatives to Animal Testing, established in 1981, began to champion the approach. Regulatory reforms, notably the United States' 1985 amendments to the Animal Welfare Act and the United Kingdom's Animals (Scientific Procedures) Act 1986, incorporated elements of the Three Rs into law. By the 1990s the framework had become the common language of research ethics committees, regulators and welfare scientists across much of the world. Interpretive drift Popularity came at a price. In their 2015 paper in the Journal of the American Association for Laboratory Animal Science, Jerrold Tannenbaum and B. Taylor Bennett examined how the definitions of the Three Rs had shifted since 1959 and argued that inconsistency in how they are defined and applied creates real confusion in practice. Some institutions define refinement narrowly, as the reduction of pain and distress during procedures; others extend it to the enhancement of positive welfare throughout an animal's life. Some treat reduction as any decrease in numbers; others insist, as Russell and Burch did, on the link to information of a given precision. Some treat replacement as applying only to methods that eliminate animal use entirely; others count the use of less sentient animals, such as invertebrates or early developmental stages, as a form of replacement. These are not merely semantic disputes. How the terms are defined determines what counts as compliance, what research gets funded as "3Rs research", and what a protocol reviewer is entitled to ask for. The modern trend, reflected in guidance from the NC3Rs and in the wording of the European Directive, is towards broader and more ambitious definitions. Refinement is increasingly understood to encompass not only the minimisation of pain, suffering, distress and lasting harm but also the promotion of positive welfare: the opportunity for animals to perform motivated behaviours, to experience comfort and to have some control over their environment. Replacement is increasingly defined as the use of methods that avoid or replace the use of animals, with the use of invertebrates or immature forms treated as a partial replacement rather than a full one. The Three Rs as a scientific discipline The most important thing to recover from Russell and Burch is their conviction that humane technique and good science are allied. They did not claim that every humane improvement would also improve the science, or that no scientifically valuable experiment would ever require animal suffering. They claimed something more modest and more robust: that inhumanity in the laboratory is, much more often than researchers assume, a symptom of poor technique, and that the effort to reduce it tends to expose and correct scientific weaknesses at the same time. The evidence gathered since 1959 has largely vindicated this view. Stressed animals show altered immune function, endocrine profiles, metabolism and behaviour, all of which can confound experimental outcomes. Animals in pain may eat less, move less and sleep poorly, with downstream effects on almost any physiological measurement. Studies with inadequate sample sizes produce unreliable results that cannot be replicated, so that the animals used in them were used for nothing. Poorly designed experiments that fail to randomise or blind are vulnerable to bias that inflates effect sizes, leading whole fields down paths that later collapse. In each case, the welfare problem and the scientific problem are the same problem seen from different sides. This convergence should not be overstated. There are genuine cases in which welfare and scientific objectives conflict: an analgesic that interferes with the inflammatory process under study, an endpoint that must be observed at a stage of disease that involves suffering, a housing refinement that introduces variation into a carefully controlled environment. These conflicts are real, and the later chapters of this book address how they should be handled. But they are fewer than is often supposed, and the default assumption, that welfare improvements cost scientific quality, is usually wrong. The more productive starting point, and the one Russell and Burch proposed, is that each welfare problem in a protocol should be examined as a possible scientific problem too. When the Rs pull in different directions Because the three principles are applied to the same animals and the same experiments, they sometimes compete, and a mature application of the framework requires judgement about how to weigh them. Three kinds of tension recur. The first is between reduction and refinement at the level of the individual animal. Longitudinal designs, in which the same animals are measured repeatedly over time, can dramatically reduce the number of animals needed, because each animal serves as its own control and between-animal variation is removed from the comparison. But repeated measurement may mean repeated anaesthesia, repeated blood sampling, repeated handling or repeated imaging, each of which carries some welfare cost. The reuse of animals across separate procedures raises the same issue in a sharper form. The European Directive addresses this explicitly: an animal that has already been used in one or more procedures may be reused only if the actual severity of the previous procedures was mild or moderate, a veterinarian has advised in favour, its general state of health and well-being has been fully restored, and the further procedure is classified as no more than moderate, subject to limited exceptions. The underlying principle is that reduction in numbers cannot be purchased with an unacceptable increase in the cumulative burden borne by the animals that remain. The second tension is between replacement and the other two Rs at the level of a research programme. Developing and validating a replacement method may itself require animal studies, for instance to generate the reference data against which the new method is compared. In the short term this can increase animal use; the case for it rests on the long-term reductions that a validated alternative will deliver. Deciding when such investment is worthwhile is a strategic judgement, and it is one reason why funders such as the NC3Rs support replacement research directly rather than leaving it to individual laboratories. The third tension concerns species choice. A study might be performed on fewer animals of a species with greater capacity for suffering, or on more animals of a species with less. Russell and Burch did not provide a formula for this, and none has been generally accepted since. Modern practice tends to prefer the species of lowest neurophysiological sensitivity that can answer the question validly, which is one reason why zebrafish larvae before the age of independent feeding, Drosophila and nematodes have become attractive for early-stage work. But validity comes first: using a less sentient organism that cannot model the relevant biology simply wastes those organisms and delays the question. These tensions are not failures of the framework. They are the reason it requires trained human judgement to apply, and the reason the law places that judgement in the hands of ethical review bodies rather than reducing it to a checklist. Beyond the original three The Three Rs sit within a wider body of thinking about animal welfare that has developed alongside them. The Brambell Committee, reporting to the UK Parliament in 1965 on the welfare of farmed animals, gave rise to what became known as the Five Freedoms: freedom from hunger and thirst, from discomfort, from pain, injury and disease, from fear and distress, and freedom to express normal behaviour. Although developed for agriculture, the Five Freedoms shaped thinking about laboratory housing and husbandry for decades. More recently, the Five Domains model developed by David Mellor and colleagues has moved the emphasis from the absence of negative states towards the presence of positive ones, assessing nutrition, physical environment, health and behavioural interactions as inputs to the animal's overall mental state. This shift, from avoiding harm towards promoting a life worth living, has been absorbed into modern interpretations of refinement. It matters for laboratory animals because many of them spend the great majority of their lives not undergoing procedures but living in cages. What happens to them in those hours, how much space and complexity they have, whether they can nest, burrow, hide, climb or interact with companions, is as much a part of their welfare as the procedure itself, and often a larger part. From principle to practice What turns this conviction into practice is a set of questions that can be asked of any proposed study. Could the question be answered without using live animals, or with fewer, or with less harm? If animals are necessary, is the chosen species and model the one that best represents the mechanism of interest, rather than the one that is most familiar or convenient? Is the experiment designed so that its results will be reliable, with appropriate controls, randomisation, blinding and sample size? Has every source of avoidable harm, direct and contingent, been identified and addressed? Are the endpoints set at the earliest point that answers the scientific question? Will the results be reported in enough detail for others to evaluate and build on them, so that the animals' use is not wasted by an irreproducible paper? None of these questions is new, and none of them is sufficient on its own. Together they constitute the working form of the Three Rs, and the rest of this book is an elaboration of how to answer them well. The regulatory frameworks described in the next chapter are, in large part, institutional machinery for making sure that these questions are asked, by someone, before any animal is used. Chapter 2: The Regulatory Architecture The Three Rs became powerful when they became law. A principle that scientists were invited to consider turned into a set of obligations that must be satisfied before a project can begin, reviewed while it runs and reported on after it ends. The legal frameworks differ considerably between jurisdictions, in what they cover, who enforces them and how much discretion they leave to institutions. But all of the major systems share a basic structure: an animal may be used in science only when a responsible body has concluded that the use is justified, that its harms have been minimised and that the people doing the work are competent to do it. This chapter sets out the main frameworks in outline. It is not a substitute for the text of the regulations or for institutional guidance, both of which change and both of which govern in detail. Its purpose is to show how each system translates the Three Rs into decisions, and where in the life of a project those decisions fall. The European Union: Directive 2010/63/EU The most comprehensive codification of the Three Rs in law is Directive 2010/63/EU on the protection of animals used for scientific purposes, adopted in September 2010 and applicable in member states from January 2013. It replaced a 1986 directive that had become outdated and inconsistently applied, and it was explicitly designed to raise and harmonise standards across the Union. The Directive states in its recitals that its ultimate goal is the full replacement of procedures on live animals for scientific and educational purposes as soon as it is scientifically possible to do so. Until then, it seeks to ensure the highest possible standard of protection for the animals that are used. Its scope is broad. It covers live non-human vertebrates, including independently feeding larval forms and foetal forms of mammals from the last third of their normal development, and live cephalopods such as octopus and squid. The inclusion of cephalopods, on the basis of evidence of their capacity to experience pain, suffering, distress and lasting harm, was a notable extension compared with most other jurisdictions. The Three Rs are written into the Directive as legal requirements rather than aspirations. Article 4 requires member states to ensure that, wherever possible, a scientifically satisfactory method or testing strategy not entailing the use of live animals is used instead of a procedure; that the number of animals used is reduced to a minimum without compromising the objectives of the project; and that breeding, accommodation and care, and the methods used in procedures, are refined so as to eliminate or reduce to a minimum any possible pain, suffering, distress or lasting harm. Article 13 adds that a procedure must not be carried out if another method not entailing the use of a live animal, and recognised under Union legislation, exists; and that in choosing between procedures, those that use the minimum number of animals, involve animals with the lowest capacity to experience pain, suffering, distress or lasting harm, cause the least of those harms and are most likely to provide satisfactory results should be selected. Death as an endpoint is to be avoided as far as possible and replaced by early and humane endpoints. Project evaluation and the harm-benefit analysis The mechanism that brings these requirements to bear on individual research is project authorisation. Under Article 36, no project may be carried out without prior authorisation by the competent authority, and authorisation depends on a favourable project evaluation. Article 38 specifies what the evaluation must include: verification that the project is justified from a scientific or educational point of view or required by law; verification that its purposes justify the use of animals; verification that it is designed so as to enable procedures to be carried out in the most humane and environmentally sensitive manner possible; an assessment of the project's compliance with the Three Rs; a classification of the severity of the procedures; and a harm-benefit analysis, to assess whether the harm to the animals in terms of suffering, pain and distress is justified by the expected outcome, taking into account ethical considerations, and may ultimately benefit human beings, animals or the environment. The harm-benefit analysis is the heart of the system and its most contested element. It requires reviewers to weigh incommensurable quantities: the suffering of particular animals against the prospect of knowledge or benefit that is uncertain, delayed and often indirect. No formula makes this weighing mechanical, and the Directive does not attempt to provide one. What it does require is that the weighing be done explicitly, by people with appropriate expertise, before the project begins. The discipline of writing down what harms are expected, what benefits are claimed and how likely those benefits are is itself a significant safeguard. It forces the applicant to confront the question of whether the study is capable of delivering what it promises, which is often where weak designs are exposed. Severity classification Article 15 requires every procedure to be classified, prospectively, into one of four categories set out in Annex VIII: non-recovery, mild, moderate or severe. The classification is based on the degree of pain, suffering, distress or lasting harm expected to be experienced by an individual animal during the course of the procedure. Annex VIII provides criteria and examples for each category, and the European Commission has published further guidance in the form of working documents on the severity assessment framework. The categories matter in several ways. They inform the harm side of the harm-benefit analysis. They determine whether a project requires retrospective assessment. They constrain reuse of animals, as described in the previous chapter. And they impose an upper limit: Article 15 provides that member states must ensure that a procedure is not performed if it involves severe pain, suffering or distress that is likely to be long-lasting and cannot be ameliorated. A safeguard clause allows a member state, in exceptional and scientifically justified circumstances, to permit such a procedure provisionally, subject to notification and review at Union level. The practical effect is that there is a ceiling on permitted suffering, not merely a requirement to justify it. The Directive also requires that the actual severity experienced by each animal be recorded and reported, not just the prospective classification. Annual statistical reports from member states therefore include the actual severity of procedures, which allows comparison between what was predicted and what happened. This retrospective reporting has had a useful disciplinary effect: it creates a record against which predictions can be checked, and it exposes procedures whose real welfare cost was higher than anticipated. Retrospective assessment, welfare bodies and transparency Article 39 requires retrospective assessment of projects that use non-human primates or involve procedures classified as severe, and allows competent authorities to require it for others. The assessment evaluates whether the objectives of the project were achieved, the harm actually inflicted on animals, including the numbers and species used and the severity of the procedures, and any elements that may contribute to further implementation of the Three Rs. Its purpose is learning: to find out whether the harm-benefit judgement made at the outset was borne out, and to feed what was learned into future projects. Each breeder, supplier and user must establish an animal-welfare body under Article 26, including at least the person responsible for welfare and care and, in the case of a user, a scientific member. The body advises staff on welfare, advises on the application of the Three Rs, establishes and reviews internal processes for monitoring and reporting, follows the development and outcome of projects, and advises on rehoming schemes. Article 25 requires each establishment to have a designated veterinarian with expertise in laboratory animal medicine. Transparency is addressed through Article 43, which requires non-technical project summaries to be published, giving the objectives of the project, predicted harms and benefits, the number and types of animals, and a demonstration of compliance with the Three Rs. The summaries are written for a general audience and are now collected in a publicly searchable European database. The United States: two overlapping systems The United States regulates animal research through two frameworks that overlap but differ in scope, legal basis and enforcement. Understanding their relationship is essential for anyone working in or with American institutions. The Animal Welfare Act The Animal Welfare Act, originally passed in 1966 as the Laboratory Animal Welfare Act and amended several times since, is a federal statute enforced by the Animal and Plant Health Inspection Service of the US Department of Agriculture. Its regulations are set out in Title 9 of the Code of Federal Regulations. The 1985 amendments, known as the Improved Standards for Laboratory Animals Act, were particularly significant for research. They required research facilities to establish Institutional Animal Care and Use Committees, required investigators to consider alternatives to procedures that may cause more than momentary or slight pain or distress, required consultation with a veterinarian in planning such procedures, required the use of anaesthetics, analgesics and tranquillisers unless withholding them is scientifically justified, and introduced requirements for the exercise of dogs and the psychological well-being of non-human primates. The Act's coverage is limited in a way that surprises many people outside the field. Its definition of "animal" excludes birds, rats of the genus Rattus and mice of the genus Mus bred for use in research, as well as cold-blooded animals and farm animals used for agricultural research. Since rats and mice bred for research make up the great majority of animals used in American laboratories, most research animals fall outside the Act's direct protection. The species it does cover include dogs, cats, non-human primates, rabbits, guinea pigs and hamsters, among others. The USDA requires registered facilities to report annually the numbers of covered animals used, classified by pain category, including whether procedures involving pain or distress were conducted with or without appropriate relief. The PHS Policy and the Guide The gap left by the Act's exclusions is largely filled, for publicly funded research, by the Public Health Service Policy on Humane Care and Use of Laboratory Animals, administered by the Office of Laboratory Animal Welfare at the National Institutes of Health. The PHS Policy applies to all live vertebrate animals used in research, research training or testing conducted or supported by PHS agencies, including the NIH. Institutions receiving such funding must file an Animal Welfare Assurance with OLAW, describing their programme for animal care and use and committing to comply with the Policy. The PHS Policy requires institutions to use the Guide for the Care and Use of Laboratory Animals, published by the National Research Council, as the basis for their programmes. The current eighth edition was published in 2011. The Guide is a substantial document covering the institutional programme, the animal environment, housing and management, veterinary care and the physical plant. It endorses the Three Rs explicitly and adopts a largely performance-based approach, specifying the outcomes that must be achieved while allowing institutions flexibility in how they achieve them. The Policy also incorporates the US Government Principles for the Utilization and Care of Vertebrate Animals Used in Testing, Research, and Training, a short set of principles issued in 1985 which, among other things, state that procedures with animals should avoid or minimise discomfort, distress and pain consistent with sound scientific practices, and that investigators should consider appropriate alternatives such as mathematical models, computer simulation and in vitro biological systems. Many institutions additionally seek voluntary accreditation from AAALAC International, which assesses programmes against the Guide and other standards and is widely regarded as a mark of quality in the United States and internationally. The IACUC Under both frameworks, the central decision-making body is the Institutional Animal Care and Use Committee. The IACUC must include at least a veterinarian with programme responsibility, a practising scientist experienced in animal research, a member whose primary concerns are in a non-scientific area, and a member not affiliated with the institution. It reviews and approves, requires modifications to, or withholds approval of proposed animal activities; conducts semi-annual reviews of the institutional programme and inspections of facilities; investigates concerns; and has the authority to suspend activities that are not being conducted in accordance with approved protocols. The IACUC's review criteria mirror the Three Rs, though they are expressed differently from the European framework. The committee must determine, among other things, that procedures avoid or minimise discomfort, distress and pain; that the investigator has considered alternatives to painful procedures and has provided a written narrative of the methods and sources used to determine that alternatives were not available; that the activities do not unnecessarily duplicate previous experiments; that appropriate sedation, analgesia or anaesthesia will be used; and that animals that would otherwise experience severe or chronic pain or distress that cannot be relieved will be painlessly euthanised at the end of the procedure or, if appropriate, during it. The requirement to justify the number of animals requested is standard in IACUC practice and embedded in the PHS Policy's requirement that the protocol describe the rationale for the number of animals. What the American system does not require, in contrast to the European one, is an explicit harm-benefit analysis with prospective severity classification in the Directive's sense. The IACUC considers justification and minimisation of harm, but its mandate is framed more around the adequacy of the welfare provisions for a given study than around a formal weighing of the study's value against its costs. How much this difference matters in practice is debated; many IACUCs engage in something close to harm-benefit reasoning, while others focus narrowly on welfare procedures and leave scientific merit to peer review. The United Kingdom The United Kingdom has regulated animal research by statute since 1876, and its current framework, the Animals (Scientific Procedures) Act 1986, was amended in 2012 to transpose the European Directive. The amended Act was retained after the UK left the European Union, so its substantive requirements remain closely aligned with the Directive, including harm-benefit analysis, severity classification, retrospective assessment and the protection of cephalopods. The British system is distinctive in its three-tier licensing structure, administered by the Home Office. An establishment licence covers the premises. A project licence authorises a programme of work and is granted only after a harm-benefit assessment. A personal licence authorises an individual to carry out regulated procedures, and is granted only to people who have completed accredited training. Each establishment must appoint named persons with defined responsibilities: a Named Veterinary Surgeon, a Named Animal Care and Welfare Officer, a Named Information Officer and a Named Training and Competency Officer, together with an Animal Welfare and Ethical Review Body that fulfils the role of the Directive's animal-welfare body and more. The Home Office's inspectorate conducts visits and audits. Non-technical summaries of all project licences are published. Table 1 sets the main features of these frameworks side by side. Table 1. Main features of three regulatory frameworks for animal research. Feature EU Directive 2010/63/EU United States (AWA and PHS Policy) United Kingdom (ASPA 1986, amended 2012) Species covered Live non-human vertebrates, some foetal and larval forms, cephalopods AWA: excludes purpose-bred rats, mice and birds; PHS: all live vertebrates in funded work As EU, including cephalopods Review body Competent authority, advised by animal-welfare body IACUC Home Office, advised by AWERB Harm-benefit analysis Explicit legal requirement Not formally required Explicit legal requirement Severity classification Prospective and actual, four categories USDA pain categories for covered species Prospective and actual, four categories Retrospective assessment Required for primates and severe procedures Not required Required on similar basis Public summaries Non-technical summaries published Not required Non-technical summaries published Recent shifts in policy toward human-based methods The regulatory landscape is moving, and in the past few years the movement has been noticeably towards encouraging or requiring non-animal methods where they are adequate. In the European Union, the prohibition on animal testing of cosmetic ingredients and finished products, fully in force for marketing purposes since 2013, created one of the first sectors in which animal testing was effectively removed, and it drove substantial investment in validated alternatives. In the United States, the FDA Modernization Act 2.0, signed into law in December 2022, removed the statutory language that had been read as requiring animal testing before human trials of new drugs, making clear that sponsors can use suitable non-animal methods, including cell-based assays, microphysiological systems and computer models, to support applications. In 2025 the FDA announced a plan to reduce and eventually replace animal testing requirements in some areas of preclinical safety assessment, beginning with monoclonal antibodies, and the NIH announced initiatives to prioritise human-based research approaches. The details and pace of these changes continue to develop, and anyone relying on them should consult the current guidance of the agencies concerned. These shifts do not remove the need for animal studies in many areas, and regulators continue to require them where no adequate alternative exists. What they change is the burden of justification. It is increasingly the animal study, rather than the alternative, that must explain why it is the right method for the question. That shift, from treating animal data as the default to treating it as one option among several, each to be justified for a given context, is the regulatory expression of the argument Russell and Burch made about models. What the law cannot do Legal frameworks set floors, not ceilings. They ensure that certain questions are asked and certain standards met, and they provide mechanisms for enforcement when things go wrong. They cannot ensure that the questions are asked thoughtfully or that the standards are embraced rather than merely satisfied. An IACUC or animal-welfare body can approve a protocol that complies with every requirement and still fails to use the best available design, the most suitable model or the most refined technique, simply because nobody involved knew of a better option or thought to look for one. This is why the Three Rs cannot be delegated entirely to compliance. The regulatory frameworks create the occasions on which improvements can be made, at the time of project design, at review, during monitoring and at retrospective assessment. Whether improvements are actually made depends on the knowledge and commitment of the people involved. The remaining chapters of this book are about that knowledge: what replacement, reduction and refinement look like in concrete terms, and how to recognise the opportunities for each. Chapter 3: Replacement in Practice Replacement is the first of the Three Rs and the most ambitious, because where it succeeds it removes the welfare problem altogether rather than mitigating it. It is also the one most often misunderstood. It is sometimes presented as a single technology waiting to be invented, a machine that will one day make animal research unnecessary. In reality it is a large and heterogeneous collection of methods, some decades old and some very new, each suited to particular questions and unsuited to others. The practical skill of replacement lies less in knowing about any one method than in knowing how to break a research question into parts, some of which can be answered without animals, and in recognising when a non-animal approach answers a question more directly than an animal study could. Kinds of replacement It helps to distinguish several kinds of replacement, because they differ in what they achieve and in how easily they can be adopted. Full replacement avoids the use of any animal or animal-derived material. It includes research on human volunteers, human tissues and cells, established cell lines, computational models and, in some classifications, non-biological physical or chemical methods. Examples range from the use of human-derived cell lines in drug screening to studies in human participants using non-invasive imaging, microdosing or carefully controlled challenge models. Partial replacement uses animals, or animal-derived material, in ways that are considered not to cause suffering, or uses organisms thought to have lower capacity for suffering. It includes the use of primary cells or tissues taken from animals killed humanely without any prior procedure, the use of invertebrates such as Drosophila melanogaster or Caenorhabditis elegans, and the use of immature forms of vertebrates, such as zebrafish embryos before the stage at which they feed independently, which are not protected under the European Directive. Whether partial replacement counts as replacement at all is a matter of definitional dispute. The NC3Rs includes it; some others prefer to treat it as a form of refinement. Whatever the label, it is often a valuable step. Russell and Burch's distinction between absolute and relative replacement captures a similar idea. What matters in practice is that the method chosen answers the scientific question at least as well as the animal study it replaces, and preferably better. Human tissues and human volunteers The most direct form of replacement uses human material or human subjects. Human tissue obtained from surgery, biopsies or post-mortem donation, with appropriate consent and ethical approval, allows researchers to study human physiology and pathology directly rather than through an animal proxy. Tissue banks and biobanks have made such material more widely available. Precision-cut tissue slices from human organs can be maintained for days, preserving cellular architecture and some functional properties, and are used in pharmacology, toxicology and disease research. Studies in human volunteers have also expanded as safer and more sensitive methods have developed. Non-invasive imaging, including functional magnetic resonance imaging, positron emission tomography and advanced ultrasound, allows the study of human physiology in health and disease without harm. Microdosing studies, in which volunteers receive very small, sub-pharmacological doses of a drug, combined with highly sensitive analytical techniques such as accelerator mass spectrometry, can provide early information on how a compound is absorbed, distributed and eliminated in humans. Controlled human infection models, in which carefully selected volunteers are deliberately exposed to well-characterised pathogens under close medical supervision, have been used to study malaria, influenza and other infections and to test vaccines, sometimes answering questions that animal models could not because of differences in host-pathogen biology. Such studies are governed by the ethics of human research, not animal research, and require their own careful justification, but where they are appropriate they can offer data of a directness that no animal model can match. Cell culture and its limits Cell culture is the workhorse of non-animal biology. Established cell lines, primary cells and, increasingly, cells derived from human induced pluripotent stem cells are used in virtually every area of biomedical research. The discovery by Shinya Yamanaka and colleagues, published in 2006 for mouse cells and in 2007 for human cells, that adult somatic cells could be reprogrammed into pluripotent stem cells by the introduction of a small number of transcription factors transformed the field. It made it possible to generate human cells of many types, including neurons, cardiomyocytes and hepatocytes, from individual donors, including patients with specific genetic diseases, without the ethical constraints associated with embryonic stem cells. Conventional two-dimensional cell culture has well-known limitations. Cells grown as monolayers on rigid plastic surfaces lose many of the properties they have in tissues: their shape, their polarity, their interactions with neighbouring cells and the extracellular matrix, and often their differentiated functions. Hepatocytes in standard culture, for instance, lose much of their drug-metabolising capacity within days. Immortalised cell lines, adapted over many generations to grow in culture, may have accumulated genetic and phenotypic changes that make them poor representatives of the tissue they came from. Misidentification and cross-contamination of cell lines have been documented problems for decades; HeLa cells in particular have contaminated many other lines. Authentication of cell lines, now expected by many journals and funders, is a basic requirement of reliable in vitro research. A further issue, sometimes overlooked in discussions of replacement, is the use of animal-derived materials in cell culture itself. Foetal bovine serum, still widely used as a culture supplement, is harvested from bovine foetuses at slaughter, raises its own welfare concerns and introduces batch-to-batch variability that undermines reproducibility. Many antibodies used in research are produced in animals, some by methods involving significant suffering; the production of monoclonal antibodies by the ascites method in mice has been largely abandoned in Europe in favour of in vitro production, and a 2020 recommendation by the EU Reference Laboratory for alternatives to animal testing (EURL ECVAM) argued that animal-derived antibodies should be replaced by non-animal-derived affinity reagents where possible. Moving to chemically defined, serum-free media and recombinant reagents is therefore both a replacement measure and a reproducibility measure. Regulatory toxicology: where replacement has advanced furthest The clearest successes of replacement have come in regulatory toxicology, the testing of chemicals, pharmaceuticals, cosmetics and other products for safety under legal requirements. This is partly because regulatory tests are standardised, so that a validated alternative can replace a specific test outright, and partly because political and legal pressure, particularly the European cosmetics testing ban and the REACH chemicals regulation, created strong incentives for alternatives. Validation is the process by which the reliability and relevance of a new method for a defined purpose are established. In Europe it is coordinated by EURL ECVAM, part of the European Commission's Joint Research Centre; in the United States, by the Interagency Coordinating Committee on the Validation of Alternative Methods (ICCVAM), supported by the National Toxicology Program's interagency centre. Internationally, the Organisation for Economic Co-operation and Development develops Test Guidelines that, once adopted, are accepted for regulatory purposes across member countries under the system of Mutual Acceptance of Data. This international acceptance is crucial: a method validated in one country but not accepted elsewhere will not displace animal tests for multinational companies. Several endpoints illustrate how replacement has proceeded. Skin irritation, historically assessed by applying substances to the shaved skin of rabbits, can now be assessed using reconstructed human epidermis models, three-dimensional cultures of human keratinocytes that form a multilayered, differentiated epidermis; OECD Test Guideline 439 describes the approach. Skin corrosion and serious eye damage and eye irritation have similar in vitro and ex vivo methods accepted, reducing reliance on the rabbit tests that were among the most publicly criticised procedures in toxicology. Skin sensitisation offers a particularly instructive case. Sensitisation, the process that leads to allergic contact dermatitis, involves a sequence of biological events: the chemical binds to skin proteins, activates keratinocytes, activates dendritic cells, and triggers proliferation of specific T cells. This sequence has been formally described as an Adverse Outcome Pathway, a structured representation of the chain of events from molecular initiating event to adverse outcome. Individual non-animal tests were developed for each of the early key events: a direct peptide reactivity assay for protein binding, keratinocyte reporter assays for keratinocyte activation, and assays measuring markers of dendritic cell activation. No single test captures the whole pathway, but combinations of them, integrated according to fixed rules, can predict sensitisation hazard. In 2021 the OECD adopted Guideline 497 on defined approaches for skin sensitisation, the first OECD guideline to describe defined combinations of non-animal methods, with fixed data interpretation procedures, as a replacement for animal tests for this endpoint. The guideline reported that the defined approaches performed at least as well as the mouse local lymph node assay when both were compared with human data. This example carries a general lesson. Replacement often does not work by substituting one non-animal test for one animal test. It works by understanding the biology well enough to break the endpoint into mechanistic components, testing each component in the system best suited to it, and integrating the results. The Adverse Outcome Pathway framework, developed under the OECD's auspices, provides a common language for doing this across many toxicological endpoints. Quality control of biological products A second area where replacement has made steady progress, with less public attention, is the testing of biological medicines and vaccines before batches are released for use. Many such products have historically required animal tests for each batch, because their potency or safety could not be adequately characterised by physical or chemical means alone. Because these tests are repeated for every batch of every product, they have accounted for large numbers of animals, and because some of them used death or severe illness as the measured outcome, they have been among the most harmful procedures in routine use. Three examples show how this is changing. Pyrogen testing, which checks injectable products for contamination by substances that cause fever, was for decades performed by injecting the product into rabbits and measuring their body temperature. The bacterial endotoxin test using lysate from the blood of horseshoe crabs replaced much of this, though it raised its own concerns about the capture and bleeding of the crabs; recombinant versions of the key clotting factor now offer an animal-free option. The monocyte activation test, which uses human blood cells to detect the release of fever-inducing signalling molecules, detects both endotoxin and non-endotoxin pyrogens, and the European Pharmacopoeia Commission decided in 2021 to phase the rabbit test out of its texts. Botulinum toxin products, used both medically and cosmetically, were long tested for potency by a lethality assay in mice; in 2011 the US Food and Drug Administration approved a cell-based potency assay developed by the manufacturer of one leading product, and other manufacturers have followed. Vaccine batch testing has increasingly moved from animal challenge tests towards a "consistency approach", in which the quality of each batch is assured by demonstrating that it was produced by a validated process and matches well-characterised reference batches on a panel of in vitro measures. These cases share a pattern. The animal test measured a crude, integrated outcome, such as fever or death. The replacement measures something more specific and mechanistically defined, often in human cells, and it is usually more precise, faster and cheaper. The obstacle to replacement was not scientific impossibility but the regulatory inertia of standardised requirements written into pharmacopoeias and product licences, which had to be changed text by text and product by product. Invertebrates and early life stages For questions that require an intact organism, with its interacting tissues, development and behaviour, invertebrates and the earliest life stages of vertebrates offer partial replacement. The fruit fly Drosophila melanogaster and the nematode Caenorhabditis elegans have been central to genetics and developmental biology for a century and half a century respectively, and many of the pathways that govern cell death, ageing, nutrient sensing and neural development were first worked out in them. Their short life cycles, low cost and genetic tractability make them powerful tools for screening genes and compounds before any vertebrate work is contemplated. Zebrafish embryos and larvae occupy an intermediate position. They are vertebrates, with organ systems broadly homologous to those of mammals, and their transparency in early development allows direct observation of organ formation, blood flow and cell behaviour. Under the European Directive, zebrafish are protected only from the stage at which they begin to feed independently, conventionally taken to be around five days after fertilisation at standard rearing temperatures, so studies confined to earlier stages fall outside the regulatory framework. This makes them attractive for screening, toxicology and developmental studies. Their use still requires justification on scientific grounds, and the welfare of the embryos is a matter of ongoing discussion, since the absence of legal protection is not the same as the absence of any capacity to suffer. But as a first step that can narrow the range of questions, compounds or doses that need to be taken into protected animals, they are an important tool. Tiered and integrated strategies Even where no single alternative can yet replace an animal test, non-animal methods can reduce and focus animal use. Integrated Approaches to Testing and Assessment combine existing information, computational predictions, in vitro data and, where still needed, targeted animal studies into a strategy in which each step informs the next. A chemical that is clearly positive in a validated in vitro assay may not need to be tested in animals at all for that endpoint. A chemical that is similar in structure to well-characterised substances may be assessed by read-across, in which data on the analogues are used to predict its properties, with animal testing reserved for cases where the prediction is uncertain. Large-scale programmes such as the United States' Toxicology in the 21st Century collaboration, known as Tox21, and the Environmental Protection Agency's ToxCast programme have screened thousands of chemicals across hundreds of high-throughput in vitro assays, generating data that can be used to prioritise chemicals for further assessment and to build predictive models. These programmes have not replaced animal toxicology wholesale, and their predictive performance for complex, systemic endpoints remains an active area of research, but they have shifted the field toward a model in which animal studies are the last step in an evidence-gathering process rather than the first. Finding alternatives before a study begins Both the European and American frameworks require researchers to consider alternatives before using animals, and the American framework requires a written narrative describing the search. In practice, the quality of these searches varies widely. A perfunctory database query with the terms "alternative" and the name of the disease is unlikely to find the relevant literature, because methods papers rarely describe themselves as alternatives. A useful search starts from the specific scientific question, breaks it into components, and asks for each component what methods exist for answering it, in any system. It draws on specialised resources, such as the EURL ECVAM database service on alternative methods, the NC3Rs resource hub, the Johns Hopkins Center for Alternatives to Animal Testing and the AWIC, the Animal Welfare Information Center of the US Department of Agriculture's National Agricultural Library, which provides guidance on conducting alternatives searches. Consultation with information specialists, experienced colleagues and the institution's 3Rs or welfare staff often reveals options that database searches miss. The search should cover all three Rs, not replacement alone. The American requirement refers to alternatives to procedures that may cause more than momentary or slight pain or distress, and "alternatives" here is understood to include refinements and reductions, such as less invasive techniques, better analgesia regimes or more efficient designs, as well as replacement methods. The cultural obstacles The limits of replacement are often attributed to the state of the science, and in many cases correctly. Whole-organism physiology, including the interactions between organ systems, the immune response, behaviour and development over a lifetime, remains beyond the reach of any current in vitro system, and questions that depend on these properties may still require animals. But there are also obstacles that have little to do with scientific capability. One is familiarity. Researchers trained in a particular animal model tend to continue using it, and laboratories organised around animal work have infrastructure, expertise and collaborative networks that make switching costly. Another is the expectation, among reviewers, editors and regulators, that animal data will be provided, sometimes regardless of whether it is the most informative evidence for the question at hand. Researchers report being asked by reviewers to add animal experiments to studies that relied on human cells or computational approaches, not because the animal data would answer a question the human data could not, but because animal validation was seen as a necessary mark of seriousness. A third obstacle is funding: developing and validating alternatives takes time and money, and the benefits may be diffuse and long-term. Addressing these obstacles is partly a matter of education and partly of institutional incentive. Funders that support the development of alternatives, journals that accept well-validated human-based studies on their own terms, and regulators that accept non-animal data where it is adequate all shift the balance. The recent policy changes described in the previous chapter are, in part, an attempt to remove the institutional presumption in favour of animal data. Whether they succeed will depend on how quickly the scientific community builds confidence in the newer methods, which is the subject of the next chapter. Hashtags: #ResearchAnimalWelfare #ThreeRsStrategy #Replacement #Reduction #Refinement #HumaneExperimentalTechnique #AnimalResearchEthics #AnimalWelfare #BioriskManagement #HarmBenefitAnalysis #SeverityAssessment #HumaneEndpoints #AnimalPainManagement #ExperimentalDesign #SampleSizePlanning #Randomization #Blinding #ARRIVEGuidelines #NonAnimalMethods #Organoids #OrganOnChip #ComputationalModels #IACUC #CultureOfCare #FutureOfAnimalWelfareScience

Download the Book (PDF): Introduction Every computational chemist knows the moment. The geometry is sensible, the basis set is adequate, the calculation converges, and the answer is wrong. Not wrong by a rounding error, but wrong in a way that matters: the spin state ordering of an iron complex is inverted, a bond dissociation curve bends upward where it should flatten, a catalytic barrier comes out ten kilocalories per mole too low. The culprit, most of the time, is electron correlation, the part of the electronic energy that a single averaged picture of the electrons cannot capture. Chemistry has spent seventy years building methods that recover it, and those methods are extraordinarily good for most molecules. For a stubborn minority they are not, and that minority includes some of the most interesting molecules in biology and industry. Quantum computing arrived in chemistry with a promise aimed directly at that minority. The argument, first sketched by Richard Feynman in 1982 and made concrete for molecules in 2005, goes like this. The exact wavefunction of a molecule lives in a space whose size grows exponentially with the number of electrons and orbitals. A classical computer must store and manipulate that space explicitly, or find a clever way to avoid it. A quantum computer built from qubits lives in the same kind of space by its nature. So a quantum computer should be able to represent a correlated molecular wavefunction that no classical computer can hold, and extract its energy efficiently. The argument is correct as far as it goes. What it leaves out is almost everything that determines whether a chemist will ever benefit from it: how the wavefunction gets prepared in the first place, how many measurements it takes to read out an energy to useful precision, what noise does to the calculation, how much error correction costs, and, above all, how good the classical competition actually is. The last decade has been an education in all of these, and the education has not always been comfortable for the people selling quantum chemistry as the first killer application. This booklet is written for computational chemists who want to understand the field on its merits. It assumes you know what a Slater determinant is, why coupled cluster is the workhorse of molecular energetics, and what an active space means. It does not assume you know anything about qubits, gates, or quantum algorithms, and it builds those ideas from the ground up in the language of electronic structure wherever possible. The argument of this book The central claim is simple to state and takes the whole book to earn. The variational quantum eigensolver, the algorithm that made quantum chemistry the flagship application of early quantum computers, has turned out to be a superb laboratory and a poor engine. It taught the field how to map molecules onto qubits, how to build chemically meaningful circuits, how to measure molecular Hamiltonians, and how to mitigate hardware noise. But its own bottlenecks, which are measurement cost, noise accumulation, and the difficulty of training its circuits, combine with steadily improving classical methods to make it very unlikely that VQE on noisy hardware will ever compute a molecular ground state that a classical computer cannot. If quantum computers deliver a genuine advantage for molecular ground states, the evidence now points to a later and narrower route: error-corrected machines running phase estimation on a specific class of strongly correlated systems, with classical methods doing most of the work around them. That is not a dismissal. A narrower claim that is true is worth more to a working chemist than a broad one that is not. Knowing exactly where the advantage could lie, and why, is what lets you read a press release, a funding call, or a paper and judge what it has actually shown. Why honesty matters here Quantum computing for chemistry has suffered from a particular kind of overstatement. Demonstrations on hydrogen and lithium hydride, molecules that a laptop solves exactly in milliseconds, were routinely presented as steps toward drug discovery. Hardware experiments were described as "beyond classical" and then reproduced on a single classical workstation within weeks. Resource estimates for landmark problems such as the iron-molybdenum cofactor of nitrogenase were quoted without the assumptions that made them meaningful. At the same time, the field has done some genuinely excellent science, and the most rigorous critiques of quantum advantage have come from inside it. The paper that most sharply questioned whether there is exponential quantum advantage in ground-state chemistry was written by researchers who also build quantum algorithms. The work showing that many quantum machine learning speedups could be matched classically, a line of results now called dequantization, came from a researcher, then an undergraduate, who set out to prove the opposite. The tensor-network simulations that brought several hardware "advantage" claims back to classical reach were done by people who respect the experiments enough to take them seriously. The picture as of late 2026 is more interesting than either the hype or the backlash suggests. Error correction has crossed thresholds that were aspirational five years ago. Superconducting processors now suppress logical errors as code size grows. Trapped-ion machines have run phase estimation for molecular hydrogen on error-corrected logical qubits. Hybrid workflows pairing quantum processors with the largest supercomputers in the world have treated iron-sulfur clusters with active spaces beyond exact diagonalization. And in each case the honest reading is the same: real engineering progress, no demonstrated advantage for a chemical question anyone could not already answer classically. How the book is built The chapters move from the chemistry to the machine, then to the algorithm, then to its limits, and finally to what a practitioner should believe and do. Chapter 1 sets out the electron correlation problem as it actually stands, including what the best classical methods can do and where they fail. Without an accurate picture of the competition, no claim of quantum advantage can be evaluated. Chapter 2 shows how a molecular Hamiltonian becomes an operator on qubits, through second quantization and fermion-to-qubit encodings, and why the number of Hamiltonian terms matters so much. Chapter 3 gives the minimum of quantum computing a chemist needs: qubits, gates, circuits, measurement, and noise, with particular attention to what "exponential" does and does not mean. Chapter 4 develops the variational quantum eigensolver in full, from the variational principle through the main families of trial circuits and the classical optimizers that drive them, using molecular hydrogen as a worked example. Chapter 5 examines the three problems that stop VQE from scaling: the measurement cost, the effect of noise and the real price of error mitigation, and the barren plateau phenomenon that makes large circuits untrainable, together with the uncomfortable recent result linking trainability to classical simulability. Chapter 6 walks through the experimental record from 2014 to 2026, reading each landmark for what it did and did not show. Chapter 7 turns to the classical side: dequantization, tensor-network simulation of quantum experiments, and the evidence that classical heuristics scale better on chemical problems than was assumed. Chapter 8 describes the fault-tolerant route through quantum phase estimation, the resource estimates for flagship molecules and how quickly they have fallen, and the state preparation problem that no amount of hardware solves by itself. Chapter 9 turns all of this into practical judgment: how to read a claim, which problems are worth watching, and what a computational chemist should be doing now. A word on units and conventions. Energies are in hartree unless otherwise stated, and "chemical accuracy" means the conventional threshold of 1 kilocalorie per mole, about 1.6 millihartree. That threshold describes agreement with experiment for energy differences; many quantum computing papers use it instead to mean agreement with an exact calculation in a given basis, which is a much weaker claim. The distinction recurs throughout the book, because it is one of the easiest ways for a modest result to sound like a large one. The field moves quickly, and some of what follows will date. The reasoning should not. Whatever hardware exists when you read this, the questions that decide whether it helps a chemist are the same: how large is the problem, how is the state prepared, how many measurements does the answer need, how are errors handled, and what does the best classical method get for the same effort. Chapter 1: The Correlation Problem, Stated Honestly Any serious claim that a quantum computer will help chemistry has to begin with a precise account of what classical computers already do well, and where they genuinely struggle. Too much writing on quantum chemistry begins instead with the exponential size of the many-electron Hilbert space, as though the entire discipline of electronic structure theory were still waiting for someone to notice the problem. It is not. The exponential wall is real, but the field has spent seven decades finding ways around it, and those ways work for the great majority of molecules chemists care about. The question for quantum computing is not whether the exact problem is hard. It is whether there are molecules that matter, for which every practical classical approximation fails, and which a quantum algorithm could treat at a cost that is actually achievable. What correlation is, and why it resists The Hartree-Fock method treats each electron as moving in the average field of all the others. Its wavefunction is a single Slater determinant, an antisymmetrized product of one-electron orbitals. For a typical closed-shell organic molecule near its equilibrium geometry, Hartree-Fock recovers more than 99 percent of the total electronic energy. The remaining fraction, the correlation energy, is small in absolute terms and enormous in chemical terms. Reaction energies, barrier heights, binding energies and spectroscopic gaps are all differences between large numbers, and the correlation energy is often of the same order as the difference being computed. A method that ignores it cannot be trusted for chemistry. It is useful to divide correlation into two kinds, even though the division is not sharp. Dynamic correlation arises from the instantaneous tendency of electrons to avoid each other, which a mean field smooths away. It is spread thinly across very many small contributions from excited configurations, and it is present in every molecule. Static, or nondynamic, correlation arises when two or more electronic configurations are nearly degenerate, so that no single determinant is even qualitatively right. It appears when bonds are stretched or broken, in diradicals, in many excited states, and above all in molecules containing several open-shell transition metal ions whose d electrons couple in intricate ways. The two kinds call for different tools. Dynamic correlation is handled well by methods that start from one good reference determinant and add excitations systematically, such as perturbation theory and coupled cluster. Static correlation needs a multiconfigurational starting point: a wavefunction that treats a chosen set of orbitals, the active space, with a flexible expansion over many determinants. The hard problems of chemistry are the ones that need both at once: a large, strongly correlated active space, and dynamic correlation on top of it. The exact problem and its size In a finite basis set, the exact answer is full configuration interaction, a linear combination of every Slater determinant that can be formed by distributing the electrons among the orbitals. Its energy is the lowest eigenvalue of the Hamiltonian matrix in that determinant basis. Nothing is approximated except the basis itself. The number of determinants is what makes full configuration interaction impossible beyond small systems. For a singlet state with equal numbers of up-spin and down-spin electrons, the count is the square of the binomial coefficient choosing half the electrons from the number of spatial orbitals. Ten electrons in ten orbitals gives about 63,500 determinants, which a laptop handles instantly. Eighteen electrons in eighteen orbitals gives about 2.4 billion, which was the practical limit of conventional complete active space methods for many years. Twenty-six in twenty-six gives about 10 to the 14th. The active space used in an influential 2017 study of the nitrogenase cofactor, 54 electrons in 54 orbitals, corresponds to roughly 4 times 10 to the 30th determinants. Storing one number per determinant at that size would exceed all the digital storage on Earth by many orders of magnitude. This is the exponential wall, and it is the entire basis of the quantum computing argument. But the size of a space says nothing about how much of it the true wavefunction actually uses, and here chemistry has been fortunate. In most molecules the ground-state wavefunction is dominated by a small number of configurations, or has a structure, such as locality or low entanglement, that can be exploited. Nearly every successful classical method works by finding and exploiting such structure. What the classical toolbox actually achieves It is worth being specific about the main families, because each defines a different kind of competition for a quantum algorithm. Table 1 summarizes their formal cost and their characteristic strengths and weaknesses. Table 1. Main classical methods for correlated electronic structure. Method Formal cost Handles well Characteristic weakness Density functional theory Roughly cubic to quartic Large systems, geometries, trends Uncontrolled errors for strong correlation CCSD(T) and local variants Seventh power (local: near linear) Single-reference dynamic correlation Fails for bond breaking and multireference cases CASSCF with perturbation theory Exponential in active space Small strongly correlated active spaces Active space limited to about 20 orbitals exactly DMRG Polynomial in bond dimension Large active spaces, quasi-linear systems Cost rises with entanglement in 3D-like systems Selected CI (HCI and relatives) Grows with determinants kept Compact multireference wavefunctions Memory limits for very large active spaces AFQMC Roughly cubic to quartic per sample Dynamic plus some static correlation Phaseless bias depends on trial wavefunction Density functional theory deserves a word first because it is the method most chemists actually use. It is not systematically improvable, and its errors for strongly correlated systems are uncontrolled, but it is cheap enough to apply to thousands of atoms and good enough, with a well-chosen functional, for a remarkable range of questions. For many industrial applications the realistic competitor to a quantum computer is not an exact method at all but a density functional calculation that is fast, familiar, and usually adequate. Coupled cluster with single, double, and perturbative triple excitations, CCSD(T), is the reference method for molecules dominated by one configuration. Its canonical cost scales as the seventh power of system size, which sounds prohibitive, but local correlation methods such as domain-based local pair natural orbital coupled cluster have brought near-linear scaling to molecules with hundreds of atoms, with errors typically well under a kilocalorie per mole relative to canonical results. When the reference determinant is good, CCSD(T) is so reliable that it serves as the benchmark against which cheaper methods are judged. Its weakness is exactly the static correlation problem. Stretch the triple bond of dinitrogen and CCSD(T) first overestimates the energy, then turns over and produces an unphysical hump or collapses below the true curve, because the single reference has become qualitatively wrong. The same failure afflicts many transition metal systems, where near-degenerate d orbitals make any single determinant a poor starting point. For static correlation, the traditional answer is the complete active space self-consistent field method, CASSCF, followed by a perturbative treatment of dynamic correlation outside the active space, as in CASPT2 or NEVPT2. It works well when the essential physics fits in roughly eighteen to twenty orbitals. Parallel implementations have pushed exact active spaces into the low twenties, but the exponential growth makes each additional pair of orbitals far more expensive than the last. The density matrix renormalization group, DMRG, changed that picture in the 2000s. It represents the wavefunction as a matrix product state, a chain of small tensors whose size, the bond dimension, controls the accuracy. For systems whose entanglement is limited, DMRG converges to near-exact results with polynomial cost, and it has been applied to active spaces of fifty to one hundred orbitals. It is the method that made a detailed classical treatment of the iron-sulfur clusters in biology possible, and it is the most important single reason that the frontier of classically tractable strong correlation has moved as far as it has. Selected configuration interaction methods, such as heat-bath configuration interaction, identify the important determinants iteratively, keep only those, and correct for the rest with perturbation theory. For many molecules a few million well-chosen determinants out of 10 to the 20th or more are enough for near-exact energies. Auxiliary-field quantum Monte Carlo, AFQMC, takes a different route, sampling the ground state stochastically with a cost per sample that scales polynomially. It controls the sign problem with a constraint defined by a trial wavefunction, which introduces a bias that is small when the trial is good. On challenging transition metal benchmarks, AFQMC with a multideterminant trial has reached accuracy comparable to the best available alternatives. These methods are complementary, and practitioners combine them: a DMRG or selected CI active space, embedded in a larger coupled cluster or density functional calculation, with AFQMC or perturbation theory for the remaining correlation. The frontier is not a single method but a toolkit, and a quantum algorithm has to beat the best combination for the problem at hand, not the worst member of it. A case study in classical persistence The chromium dimer shows how the classical frontier actually moves. Two chromium atoms, each with six valence electrons in 3d and 4s orbitals, form a formal sextuple bond. The molecule looks trivially small, and for decades it was a notorious failure of quantum chemistry. Its potential energy curve has an unusual shoulder at intermediate bond lengths, and method after method produced curves that were qualitatively wrong: coupled cluster collapsed, multireference perturbation theory depended sensitively on the active space, and density functionals scattered widely. Chromium dimer became a standard test that the field used to show how hard strong correlation could be, and it was repeatedly cited as the kind of problem where only an exact solver would settle the question. It was settled classically. In 2022 a team using a combination of large-scale DMRG and selected configuration interaction, with careful extrapolation and perturbative corrections for dynamic correlation outside the active space, produced a potential energy curve in close agreement with the experimental shape and binding energy, and described the work as closing a chapter of quantum chemistry. The calculation was expensive and required expert judgment at every stage, but it did not require a new kind of computer. The lesson generalizes. Problems that are held up as intractable at one moment often become tractable through algorithmic improvement, and the rate of that improvement in classical electronic structure has been high. Anyone arguing that a specific molecule needs a quantum computer has to argue not only that current classical methods fail for it but that they will keep failing over the decade it would take to build the quantum machine. Where classical methods genuinely struggle Given all this, what is left? The honest list is shorter than the quantum computing literature often implies, but it is not empty. The first category is multinuclear transition metal clusters with many strongly coupled open-shell centers. The iron-molybdenum cofactor of nitrogenase, often called FeMoco, is the celebrated example. It contains seven iron atoms, one molybdenum, nine sulfurs and a central carbon, and it catalyzes the reduction of dinitrogen to ammonia at ambient conditions, a reaction that industrial chemistry performs at high temperature and pressure through the Haber-Bosch process. Understanding its mechanism requires resolving many close-lying spin states and oxidation states, and the relevant energy differences are small. Active spaces thought to capture the essential physics run from fifty to more than one hundred orbitals. DMRG calculations have been performed at this scale, but whether they are converged, and whether the chosen active space and model are chemically adequate, remains debated. Photosystem II's manganese-calcium cluster, cytochrome P450's iron-porphyrin chemistry, and various synthetic multimetallic catalysts belong to the same family. The second category is systems where static and dynamic correlation are both large and entangled with each other, so that splitting the problem into an active space plus a perturbative correction is unreliable. Certain bond-breaking reactions on metal surfaces, some lanthanide and actinide complexes, and strongly correlated extended materials fall here. For materials in particular, such as the doped copper oxide superconductors, the physics is often captured by lattice models like the Hubbard model, which have their own well-developed classical toolkit and their own open questions. The third category is not about ground states at all but about dynamics and excited states: real-time electron dynamics, nonadiabatic processes and spectroscopy. This booklet is about ground states, but it is worth noting that some of the strongest theoretical arguments for quantum advantage concern time evolution rather than ground-state energies, because there is no variational principle for dynamics and classical methods have fewer tricks to exploit. Even within these categories, it is important to be precise about what "struggle" means. Classical methods rarely fail to produce any answer. They produce answers whose error bars are uncertain, whose convergence is hard to demonstrate, or whose cost grows steeply enough that a converged answer is out of reach for the specific molecule and active space the chemist wants. A quantum advantage in this setting would not look like a quantum computer solving a problem that classical methods cannot touch. It would look like a quantum computer delivering a more certain answer, faster or more cheaply, on a problem where classical estimates disagree. What a useful answer requires The last piece of groundwork concerns what counts as a useful answer, because it shapes the precision a quantum computer must achieve. Chemistry almost never needs total energies. It needs energy differences: between reactants and products, between transition states and minima, between spin states. These differences are typically in the range of a few to a few tens of kilocalories per mole. To be useful, a calculation generally needs to get them right to about one kilocalorie per mole, 1.6 millihartree, the conventional threshold of chemical accuracy. Some questions, such as spin-state ordering in iron complexes, demand even finer resolution. This requirement has three consequences for quantum computing. First, a calculation that reproduces the exact energy in a minimal basis set is not chemically accurate in any meaningful sense, because the basis set error in a minimal basis is often tens or hundreds of millihartree. Agreement with full configuration interaction in the same small basis tests the quantum algorithm, not the chemistry. Second, the precision required for total energies is set by the energy differences of interest, and because errors do not always cancel between different geometries or spin states, each energy must usually be computed to better than the target difference. Third, and most importantly for later chapters, the cost of many quantum algorithms scales with the inverse of the precision, or its square. Tightening the target from ten millihartree to one millihartree can multiply the number of measurements a hundredfold. There is also the matter of the model. A quantum computer, like any exact solver, computes the exact answer within an active space and basis that a human or a classical procedure has chosen. If the active space omits orbitals that matter, or the embedding into the surrounding molecule and solvent is poor, an exact solution of the wrong model is still wrong. This is one reason that realistic proposals for quantum chemistry now describe hybrid workflows in which the quantum computer solves a carefully constructed active space problem while classical methods build the model, treat the environment, and add dynamic correlation from outside the active space. The shape of the opportunity Put together, these observations define a narrow but real target. A quantum computer helps a chemist when it can solve a strongly correlated active space that is too large or too entangled for DMRG, selected CI, or AFQMC to converge with confidence; when that active space, embedded properly, answers a chemical question that matters; and when the quantum calculation reaches the needed precision at a cost, in time and money, that competes with pushing the classical methods harder. Each condition narrows the field. Many strongly correlated active spaces that sound intimidating are in fact within reach of modern DMRG. Many chemically important questions can be answered by density functional theory with a sensible functional, or by coupled cluster, without ever needing a multireference solver. And the cost of a quantum calculation, as later chapters show, depends heavily on precision, measurement strategy, and error correction overhead. None of this means the opportunity is illusory. It means it lives where classical methods are weakest, and that its size can only be judged by specific comparison on specific molecules. The rest of the book follows that principle: every quantum method is measured against what the best classical method achieves for the same problem, and every demonstration is read for what it proves about that comparison. The next step is to see how the chemist's Hamiltonian becomes something a quantum computer can act on, because the choices made in that translation shape everything that follows. Chapter 2: From Orbitals to Qubits A quantum computer does not know what an electron is. It manipulates qubits, two-level systems with no charge, no spin in the chemical sense, and no antisymmetry. Before any algorithm can compute a molecular energy, the electronic Hamiltonian must be rewritten as an operator acting on qubits. This translation looks like bookkeeping, and much of it is, but the choices made here decide how many qubits a problem needs, how long the circuits are, and how many measurements the energy will cost. Several of the most important advances in quantum chemistry algorithms over the past decade have been improvements in this translation rather than in the algorithms that use it. Second quantization in brief Computational chemists usually meet the molecular Hamiltonian in first quantization, as a sum of kinetic energy operators, electron-nuclear attractions and electron-electron repulsions written in terms of electron coordinates. The antisymmetry of the wavefunction is imposed by writing it as a Slater determinant or a combination of them. Second quantization moves the antisymmetry out of the wavefunction and into the operators. One starts from a finite set of spin orbitals, typically the canonical Hartree-Fock orbitals or some localized variant. Each spin orbital is either occupied or empty, so any Slater determinant is fully described by a string of zeros and ones, its occupation number vector. For each spin orbital there is a creation operator, which places an electron in that orbital if it is empty, and an annihilation operator, which removes one if it is present. These operators anticommute: swapping the order of two creation operators on different orbitals changes the sign of the result. That single algebraic rule enforces the Pauli exclusion principle and the antisymmetry of every state built from them. In this language, the electronic Hamiltonian within the chosen orbital basis takes a compact form. It is a sum of one-electron terms, each a coefficient times a creation operator on one orbital and an annihilation operator on another, plus two-electron terms, each a coefficient times two creation and two annihilation operators. The one-electron coefficients are integrals of the kinetic and nuclear attraction operators over pairs of orbitals. The two-electron coefficients are the familiar electron repulsion integrals over four orbitals. Any standard quantum chemistry package, such as PySCF, Psi4 or Molpro, computes these integrals routinely, and they are the only molecular input a quantum algorithm needs. The number of two-electron integrals grows as the fourth power of the number of orbitals. With symmetry and the vanishing of integrals between distant orbitals the practical number is smaller, but the fourth-power growth is the natural scale of the problem. This matters because, as later chapters show, several quantum algorithms have costs that scale with the number of terms in the Hamiltonian or with the sum of the absolute values of their coefficients. The fermion-to-qubit problem Occupation number vectors look tailor-made for qubits. A qubit has two basis states, conventionally labeled zero and one, and a register of qubits has basis states labeled by bit strings. Map each spin orbital to one qubit, and let the qubit's state record whether the orbital is occupied. A Slater determinant becomes a computational basis state of the qubit register. The Hartree-Fock determinant of a molecule with a given number of electrons is simply the bit string with ones in the lowest occupied orbitals and zeros elsewhere. The difficulty is the operators. Qubits do not anticommute. Operators on different qubits commute with each other, whereas fermionic operators on different orbitals anticommute. A faithful mapping must reproduce the fermionic sign structure using only qubit operations. Three standard encodings do this in different ways, and their trade-offs are summarized in Table 2. Table 2. Common fermion-to-qubit encodings. Encoding Stores in each qubit Operator weight Main advantage Main drawback Jordan-Wigner Occupation of one orbital Up to number of qubits Simple and transparent Long strings of Z operators Parity Parity of all orbitals up to it Up to number of qubits Two qubits removable by symmetry Long strings for creation operators Bravyi-Kitaev Partial sums of occupations Logarithmic in qubits Shorter operators at scale Less intuitive, gains modest for small molecules The Jordan-Wigner transformation, dating to 1928 and originally developed for spin chains, is the most transparent. Each qubit stores the occupation of one spin orbital. A creation or annihilation operator on orbital number k becomes a raising or lowering operation on qubit k, multiplied by a string of Pauli Z operators on all qubits with lower index. The Z string counts, with a sign, how many occupied orbitals precede orbital k, which is exactly what the anticommutation rule requires. The price is that an operator involving a high-index orbital acts nontrivially on many qubits, and on hardware where qubits interact only with their neighbors, long strings translate into long circuits. The parity encoding makes the opposite choice. Each qubit stores the parity, even or odd, of the total occupation of all orbitals up to and including its own. Parity information that the Jordan-Wigner scheme must compute with a long Z string is now available locally, but updating an occupation requires flipping every later qubit, so the long strings reappear elsewhere. Its practical virtue is symmetry reduction: the last qubit stores the total electron number parity, and with a suitable ordering another qubit stores the parity of one spin species. Both are fixed for a given molecule, so two qubits can be removed from the problem with no loss. The Bravyi-Kitaev encoding, introduced in 2002 and brought into chemistry around 2012, interpolates between the two, storing partial sums of occupations arranged on a binary tree. Both occupation and parity information then require acting on only a logarithmic number of qubits. For large systems the reduction in operator weight is significant in principle. For the small molecules that have actually been run on hardware, the difference is usually modest, and the choice of encoding is often driven by compatibility with the circuit design rather than asymptotic scaling. A worked example: molecular hydrogen The standard first example is the hydrogen molecule in the minimal STO-3G basis. Each hydrogen contributes one 1s function, giving two spatial molecular orbitals, the bonding and antibonding combinations, and four spin orbitals. Under the Jordan-Wigner transformation, four spin orbitals become four qubits, and the Hartree-Fock state, with both electrons in the bonding orbital, is the bit string with ones on the two bonding spin orbitals. The Hamiltonian becomes a weighted sum of fifteen Pauli strings, counting the identity, each a product of single-qubit Pauli operators X, Y, Z or the identity on each of the four qubits. The coefficients depend on the bond length through the integrals. The energy at any bond length is the lowest eigenvalue of this 16 by 16 matrix, restricted to states with two electrons and zero spin projection. Using symmetries, including electron number, spin, and the spatial symmetry that forbids single excitations between orbitals of different parity, the problem can be reduced to two qubits, and even to one: the only determinants that mix in the singlet ground state are the doubly occupied bonding configuration and the doubly occupied antibonding configuration, so a single rotation angle describes the exact answer. That reduction is illuminating in two ways. It shows how much of a naive qubit count can be removed by symmetry, which matters for fitting problems onto small devices. And it shows why hydrogen in a minimal basis is a test of plumbing, not chemistry. It has one parameter, a two-by-two eigenvalue problem, and an answer that is known exactly in closed form. Every early hardware demonstration was, at heart, a demonstration that the plumbing worked. Qubit counts in realistic problems With one qubit per spin orbital, qubit count is twice the number of spatial orbitals in the active space. A strongly correlated active space of 50 spatial orbitals needs 100 qubits before symmetry reduction. The full cofactor models used for nitrogenase, with 54 or 76 active orbitals, need 108 or 152 system qubits. These numbers are sometimes quoted as though they were the size of the quantum computer required. They are not: they count logical system qubits only, and any practical algorithm needs additional qubits for ancillas, for data loading, and above all for error correction, which can multiply physical qubit counts by factors of hundreds or thousands. The basis set question is where many demonstrations quietly part company with chemistry. A calculation in a minimal basis captures qualitative bonding but not quantitative energetics. Chemically useful results typically need at least a correlation-consistent double-zeta basis and often triple-zeta, with the number of orbitals growing rapidly. Hydrogen in cc-pVDZ has ten spatial orbitals, twenty spin orbitals, and already needs twenty qubits. A modest organic molecule in a triple-zeta basis has hundreds of orbitals. No quantum algorithm treats all of them directly. Instead, as with classical multireference methods, the quantum computer treats an active space, and the orbitals outside it are handled by classical methods: frozen, treated perturbatively, or folded in through embedding schemes or transcorrelated Hamiltonians that absorb some dynamic correlation into a modified operator. This makes the quantum computer an active-space solver, a drop-in replacement for the configuration interaction or DMRG step in a CASSCF-style calculation. It is a useful way to think about the whole enterprise, because it immediately identifies the classical competitor: the best available active-space solver for the same active space. It also identifies a hidden cost. Orbital optimization in CASSCF requires not just the energy but the one- and two-particle reduced density matrices of the active-space wavefunction, and extracting those from a quantum computer requires many more measurements than the energy alone. Symmetry, tapering and conserved quantities The hydrogen example hinted at a general tool. Molecular Hamiltonians commute with several symmetry operators: total electron number, the number of electrons of each spin, total spin, and the point group of the nuclear framework. Each conserved quantity divides the full space of qubit states into sectors, only one of which contains the state of interest. A calculation that wanders into the wrong sector, for instance one that ends up with three electrons in a two-electron problem because of hardware noise, produces a meaningless energy. Some of these symmetries appear, after the qubit mapping, as products of Pauli Z operators that commute with every term of the Hamiltonian. A procedure introduced in 2017, often called qubit tapering, finds such symmetries automatically and applies a transformation that turns each one into a single-qubit operator. That qubit's value is then fixed by the chosen symmetry sector, and it can be deleted. For small molecules with high point-group symmetry, tapering removes several qubits at no cost in accuracy, which is why published experiments sometimes treat a molecule on fewer qubits than its spin orbital count would suggest. Symmetries that cannot be tapered away can still be protected in other ways. Circuits can be built entirely from gates that conserve particle number and spin projection, so that the state never leaves the right sector in the absence of noise. When noise does push it out, measuring the conserved quantity and discarding runs that violate it, a technique called symmetry verification or postselection, removes a portion of the errors at the cost of throwing away data. That trade, fewer errors for more measurements, is typical of what the next chapters call error mitigation. Choosing and ordering orbitals The orbitals used in the mapping are not dictated by the molecule. Canonical Hartree-Fock orbitals are delocalized over the whole molecule. Localized orbitals, obtained by rotations such as the Pipek-Mezey or Foster-Boys procedures, concentrate each orbital on a few atoms. Natural orbitals, the eigenfunctions of a correlated one-particle density matrix, order themselves by occupation and give the most compact configuration interaction expansions. Every choice describes the same Hamiltonian, but they distribute its complexity differently. For quantum algorithms this matters in two ways. Under the Jordan-Wigner transformation, the length of each Pauli Z string depends on the order in which orbitals are assigned to qubits, so an ordering that places strongly interacting orbitals next to each other shortens circuits, particularly on hardware where only neighboring qubits interact. And the choice of orbitals determines how much of the correlation must be described by the quantum state at all: a good orbital basis puts the wavefunction closer to a simple starting point and reduces the work the quantum circuit has to do. DMRG practitioners know both effects well, because the same orbital ordering and localization choices govern the bond dimension a matrix product state needs. It is another instance of classical expertise transferring directly to the quantum setting. Factorizing the Hamiltonian Because so many costs scale with the number and size of Hamiltonian terms, a large body of work has focused on rewriting the two-electron part more efficiently. The idea is borrowed from classical quantum chemistry, where the Cholesky decomposition and density fitting of the electron repulsion integrals have been standard for decades. The four-index tensor of two-electron integrals can be viewed as a matrix between pairs of orbital indices, and that matrix is typically of low rank: its eigenvalues fall off quickly, so a truncated decomposition reproduces it well. A single factorization expresses the two-electron operator as a sum of squares of one-body operators. A double factorization goes further, diagonalizing each of those one-body operators so that each term becomes, in a rotated orbital basis, a simple function of occupation numbers. On a quantum computer, orbital rotations can be implemented efficiently with circuits of Givens rotations, and occupation-number functions are cheap to evaluate or measure. The result is a Hamiltonian expressed as a modest number of groups, each measurable in a single rotated basis or simulable with a single structured circuit. Tensor hypercontraction, also borrowed from classical electronic structure, pushes the compression further by approximating the integral tensor with a product of smaller factors. In 2021 a Google-led team used it to produce what were then the lowest published resource estimates for fault-tolerant simulations of the nitrogenase cofactor, a result discussed in Chapter 8. These methods matter for near-term algorithms as well, because they reduce the number of distinct measurement settings a variational calculation needs. First quantization as an alternative Everything above assumes a second-quantized description with a fixed orbital basis, which is natural for chemists and dominant in near-term work. There is an alternative. In first quantization, each electron is assigned a register of qubits that stores its position on a grid, or its index in a large basis set such as plane waves. The number of qubits then scales with the number of electrons times the logarithm of the basis size, rather than with the basis size itself. Antisymmetry must be imposed on the initial state, but the Hamiltonian operations preserve it thereafter. For very large basis sets, first quantization can be dramatically more economical in qubits, and plane waves avoid the basis set incompleteness of Gaussian orbitals in a systematic way. Resource estimates for fault-tolerant simulation in first quantization have been published for materials and for molecules in the plane-wave basis, and they suggest this route may be preferred for some problems once large error-corrected machines exist. For near-term devices, the additional arithmetic required makes it impractical, and nearly all experimental work has used second quantization. What the mapping teaches Three lessons from this chapter recur throughout the book. First, the qubit count of a chemical problem is set by the active space, not the molecule, and headline qubit counts are always a statement about a chosen model. Second, the structure of the Hamiltonian, meaning its number of terms, their magnitudes, and how they can be grouped or factorized, directly determines the cost of both near-term and fault-tolerant algorithms; improvements in that structure have repeatedly produced larger practical gains than improvements in hardware. Third, the translation is where chemistry and quantum information meet most concretely, and it rewards chemists who understand both. The same integrals, the same orbital choices and the same symmetries that make classical methods efficient make quantum methods efficient too. With the Hamiltonian expressed as an operator on qubits, the next question is what a quantum computer can do with it. That requires a short but careful account of the machine itself. Chapter 3: The Machine, for Chemists A chemist does not need to understand superconducting circuit design or ion trap optics to evaluate quantum chemistry algorithms. But a few features of how quantum computers work are decisive for everything that follows, and they are often explained either too loosely to be useful or too formally to be read. This chapter gives the minimum, framed wherever possible in terms a computational chemist already uses. States, amplitudes and the catch A single qubit is a two-level quantum system. Its state is a superposition of the two basis states, zero and one, described by two complex amplitudes whose squared magnitudes sum to one. So far this is familiar: it is the same mathematics as a spin one-half particle, or a two-state configuration interaction problem. A register of n qubits has two to the n basis states, one for every bit string, and its general state is a superposition over all of them, described by two to the n complex amplitudes. Under the Jordan-Wigner mapping of the last chapter, those bit strings are Slater determinants and those amplitudes are configuration interaction coefficients. A register of 100 qubits can therefore hold a wavefunction over 2 to the 100th determinants, about 10 to the 30th, which is precisely the full configuration interaction space of a large strongly correlated active space. This is the central attraction: the quantum computer does not store the wavefunction as a list of numbers; it physically is the wavefunction. The catch is equally central. The amplitudes cannot be read out. When a qubit register is measured in the computational basis, the result is a single bit string, drawn at random with probability equal to the squared magnitude of its amplitude, and the superposition is destroyed. To learn anything about the state, one must prepare it again and measure again, many times, and infer properties from the statistics. A quantum computer that holds a perfect full configuration interaction wavefunction for FeMoco cannot simply print the coefficients. It can only be sampled. For chemistry this means the output of any quantum algorithm must be a quantity that can be estimated efficiently from samples: an energy, a dipole moment, a reduced density matrix element, or a phase. Many useful quantities qualify. But the number of samples needed to reach a given precision becomes a first-order cost, and it is where near-term algorithms most often run into trouble. Gates and circuits Quantum computations are built from gates, elementary unitary operations on one or two qubits, arranged in sequence into circuits. Single-qubit gates rotate a qubit's state; two-qubit gates, such as the controlled-NOT, entangle pairs. A small set of such gates is universal, meaning that any unitary operation on the register can be approximated to arbitrary accuracy by a sequence of them. Universality says nothing about efficiency. Most unitaries on n qubits require a number of gates that grows exponentially with n. The useful ones, those that implement a quantum algorithm with a polynomial number of gates, have special structure. For chemistry, the most important structured operations are orbital rotations, which transform one set of orbitals into another and can be implemented with circuits of Givens rotations whose depth grows linearly with the number of orbitals; time evolution under the molecular Hamiltonian, which can be approximated by products of exponentials of individual terms; and exponentials of excitation operators, the quantum counterparts of the excitation operators in coupled cluster theory. Two properties of a circuit dominate its practical cost. The gate count, particularly the number of two-qubit gates, determines how much error accumulates, because two-qubit gates are typically an order of magnitude noisier than single-qubit ones. The depth, the number of sequential layers of gates, determines how long the qubits must hold their state, and so how much they decohere. Both depend on hardware connectivity. If only neighboring qubits can interact directly, a two-qubit gate between distant qubits must be built from a chain of swaps, inflating both count and depth. A small circuit, read as chemistry It helps to trace one tiny circuit in chemical terms. Take the four-qubit hydrogen problem from the last chapter, with qubits zero and one representing the up and down spin orbitals of the bonding orbital, and qubits two and three the antibonding ones. All qubits start in the zero state, the empty vacuum. Applying a bit-flip gate, the Pauli X, to qubits zero and one produces the bit string with ones in the bonding spin orbitals: the Hartree-Fock determinant, prepared exactly with two single-qubit gates. Now apply a structured sequence of two-qubit gates that implements a rotation by some angle between the doubly occupied bonding configuration and the doubly occupied antibonding configuration. The register now holds a superposition of two determinants, with coefficients equal to the cosine and sine of that angle. This is a two-configuration wavefunction of exactly the kind a chemist would write down to describe a stretched hydrogen bond, and for hydrogen in this basis it contains the exact ground state for a suitable angle. Measuring the register in the computational basis returns either the Hartree-Fock bit string or the doubly excited one, with probabilities given by the squared coefficients. Averaging many such measurements gives the expectation value of any Hamiltonian term built only from Z operators, which in the Jordan-Wigner picture correspond to orbital occupations and their products. The Hamiltonian also contains terms with X and Y operators, which correspond to the exchange-like processes that move electrons between orbitals. Those cannot be read off in the computational basis. Before measuring them, one applies single-qubit rotations that change the measurement basis, so that each such term becomes diagonal. Different groups of terms need different rotations, and so different runs of the circuit. That small example contains the whole structure of the variational approach in miniature: prepare a reference, apply a parameterized entangling circuit, measure in several bases, and assemble an energy from the averages. It also contains its principal cost, which is that every expectation value comes from repeated preparation and measurement. Today's hardware Three physical platforms dominate current machines, and their differences matter for chemistry. Superconducting qubits are small electrical circuits cooled to about ten millikelvin. Gates take tens of nanoseconds, so circuits run quickly, but each qubit connects only to a few neighbors on a fixed chip layout. Google's Willow processor, announced in December 2024, has 105 qubits. IBM's Heron processors, with 133 and later 156 qubits, have been the workhorses of its cloud service, and in November 2025 IBM introduced Nighthawk, a 120-qubit chip with a square lattice in which each qubit couples to four neighbors, designed to support circuits with around 5,000 two-qubit gates. Trapped-ion qubits are individual charged atoms held in electromagnetic traps and manipulated with lasers. Gates are slower, typically tens to hundreds of microseconds, but fidelities are the highest of any platform and ions can be shuttled so that any pair can interact. Quantinuum's Helios system, launched commercially in November 2025 and described in Nature in 2026, holds 98 barium ions with all-to-all connectivity, and the published characterization reports average two-qubit gate infidelities of about 8 in 10,000, a fidelity near 99.92 percent. Neutral atom machines hold uncharged atoms in arrays of optical tweezers and entangle them by exciting them to high-energy Rydberg states. They scale in qubit number more easily than the other platforms, with arrays of thousands of atoms demonstrated, and atoms can be rearranged during a computation. Their gate fidelities have improved rapidly, and some of the most ambitious logical qubit experiments of the past few years have been done on this platform. A fair summary for chemists is this: the best physical two-qubit gates now fail somewhere between one time in a thousand and one time in a few hundred, depending on platform and operating conditions, and the largest machines have between about one hundred and a few thousand physical qubits. Neither number, by itself, says whether a chemical calculation is feasible. What matters is how many gates the calculation needs relative to the inverse error rate, and how the errors are handled. Why classical simulators run out The exponential size of the state space explains why quantum circuits cannot simply be simulated classically at scale, and it also explains where the line sits. A statevector simulator stores all two to the n complex amplitudes, each taking sixteen bytes in double precision. Thirty qubits need about 17 gigabytes, within reach of a workstation. Forty qubits need about 17 terabytes, which requires a large cluster. Fifty qubits would need about 18 petabytes, more memory than any existing supercomputer possesses. Brute-force simulation therefore stops somewhere in the mid-forties of qubits. That boundary is often quoted as the point beyond which a quantum computer does something classically impossible. It is not. Tensor-network simulators do not store the full state; they store a compressed representation whose size depends on how much entanglement the circuit creates, and for circuits with limited entanglement they can simulate hundreds of qubits. Other methods propagate observables rather than states and exploit the damping effect of noise. The relevant boundary is not the qubit count at which brute force fails but the circuit complexity at which every clever classical method fails, and that boundary is much harder to locate. It is the same distinction as the one between exact diagonalization and DMRG in electronic structure, and it will recur throughout the book. Noise and the error budget Every gate, every idle moment and every measurement introduces a small probability of error. Errors accumulate. A useful rule of thumb is that a circuit can reliably execute roughly as many two-qubit gates as the inverse of the two-qubit error rate before the output is dominated by noise. With an error rate of one in a thousand, that is on the order of a thousand gates, spread across all the qubits. More gates than that and the probability that the whole circuit ran without any error becomes small, and the computed quantity drifts toward the value it would have in a random state. For chemistry this is a severe constraint. A chemically meaningful ansatz for an active space of even twenty orbitals, built from coupled-cluster-style excitation operators and compiled for hardware with limited connectivity, can easily need tens of thousands of two-qubit gates or more. Time evolution for phase estimation on problems of interest needs many orders of magnitude more. John Preskill named the current period the era of noisy intermediate-scale quantum devices, NISQ, in 2018, precisely to mark the gap between what these devices can hold and what they can reliably compute. There are two responses to noise. Error mitigation accepts that the hardware is noisy and uses classical post-processing of carefully designed extra experiments to estimate what the noiseless result would have been. It needs no extra qubits, but its cost in samples grows, in general exponentially, with the size of the circuit. Error correction encodes each logical qubit in many physical qubits, detects errors as they occur by measuring collective properties of the physical qubits, and corrects them before they spread. It needs large overheads in qubits and time, but it lets computations run arbitrarily long, provided the physical error rate is below a threshold. Chapter 5 examines mitigation and Chapter 8 examines correction; the distinction between them is the most important dividing line in the field. What "exponential" does and does not mean Popular accounts often say that a quantum computer with n qubits performs two to the n calculations at once. That description is misleading, and for chemists it leads to a specific error: the belief that because a quantum register can hold a full configuration interaction wavefunction, a quantum computer can find the ground state of any molecule efficiently. It cannot. The relevant result comes from computational complexity theory. Finding the ground-state energy of a general local Hamiltonian, even to modest precision, is QMA-complete, the quantum analogue of NP-complete. It is believed that no efficient algorithm, classical or quantum, solves it in the worst case. A 2009 result showed that the electronic structure problem in its general form, electrons in an arbitrary external potential, inherits this hardness. So a quantum computer does not make ground-state chemistry easy in general, any more than a classical computer makes the traveling salesman problem easy. What a quantum computer can do efficiently is somewhat different. Given a Hamiltonian and an initial state that has a reasonable overlap with the true ground state, quantum phase estimation projects onto the ground state with probability equal to the squared overlap and returns its energy to a precision that improves linearly with the length of the computation. The hard part, finding the ground state, has been converted into a different hard part, preparing a state that already resembles it. If a simple state such as Hartree-Fock has large overlap with the true ground state, the problem is easy. If the overlap is exponentially small, the procedure takes exponentially many repetitions. This reframing is essential for everything that follows. The case for quantum advantage in ground-state chemistry rests on the existence of molecules in a middle ground: hard enough that classical methods cannot solve them with confidence, but structured enough that a good starting state can be prepared on a quantum computer. Whether such molecules exist, and how many, is not a question complexity theory answers. It is an empirical question about chemistry, and it is at the center of Chapter 7. Sampling and precision Because every useful output is estimated from repeated measurement, statistical precision follows the familiar rule of Monte Carlo methods. The standard error of an average over N independent samples falls as one over the square root of N. Halving the error requires four times as many samples; improving it tenfold requires a hundred times as many. For an energy computed by measuring each Hamiltonian term separately, the number of samples required to reach a target precision depends on the sum of the variances of the terms, which in turn depends on the magnitudes of the Hamiltonian coefficients. Molecular Hamiltonians have many terms, and some of their coefficients are large. The result, examined quantitatively in Chapter 5, is that estimating a molecular energy to one millihartree by direct sampling can require an enormous number of circuit repetitions, and that this cost, not the qubit count, is often the binding constraint on variational algorithms. Phase estimation escapes this particular scaling. Its precision improves as one over the total evolution time rather than one over the square root of the number of samples, a quadratic improvement known as the Heisenberg limit. That improvement is one of the main reasons the fault-tolerant route looks more promising than the variational one for problems that need chemical accuracy. It comes at the cost of circuits far deeper than any noisy device can run. Logical and physical qubits A final distinction is between physical and logical qubits. A physical qubit is a single ion, atom or superconducting circuit. A logical qubit is an error-corrected qubit encoded in many physical ones. The most studied code, the surface code, uses a two-dimensional patch of physical qubits whose size grows with the code distance; the logical error rate falls exponentially with distance provided the physical error rate is below the threshold. In December 2024 Google reported in Nature a surface code memory on its Willow processor in which the logical error rate fell by roughly a factor of two each time the code distance increased by two, from distance three to distance five to distance seven. This was the first convincing demonstration of below-threshold scaling in a superconducting surface code, and it is a genuine milestone. It was a memory experiment, preserving one logical qubit, not a computation, and the logical error rate at distance seven, around one in a thousand per cycle, is still far above what long chemistry algorithms need. Trapped-ion and neutral atom groups have demonstrated small numbers of logical qubits performing logical operations, and the first chemistry calculations on logical qubits, described in Chapter 6, have been run. The practical meaning for chemists is that resource estimates for fault-tolerant chemistry are quoted in logical qubits and logical operations, and translate into physical qubits through code overheads that depend on physical error rates, code choice and architecture. An algorithm needing a few thousand logical qubits may need a few million physical qubits under surface code assumptions, or fewer with newer codes and hardware. Those numbers, and how quickly they have been falling, are the subject of Chapter 8. First, though, comes the algorithm that was supposed to make all of this unnecessary for the near term. Hashtags: #QuantumComputingForComputationalChemists #QuantumChemistry #ComputationalChemistry #ElectronicStructure #ElectronCorrelation #MolecularGroundStates #VariationalQuantumEigensolver #VQE #QuantumPhaseEstimation #StrongCorrelation #ActiveSpaceMethods #FermionToQubitMapping #JordanWignerTransformation #BravyiKitaevTransformation #QubitTapering #MolecularHamiltonians #QuantumCircuits #ErrorMitigation #QuantumErrorCorrection #FaultTolerantQuantumComputing #StatePreparation #MeasurementCost #ChemicalAccuracy #QuantumAdvantage #FutureOfQuantumChemistry Pasted markdown

Download the Book (PDF): Introduction A researcher sits down to study how people who have been through a particular illness talk to one another. Twenty years ago this would have meant finding a support group, negotiating access with the facilitator, sitting in a circle for eight months, and writing fieldnotes on the drive home. Today it means something stranger. The group exists, but it exists as a forum with fourteen thousand members, a private messaging channel that splintered off from the forum after an argument in 2019, three overlapping hashtag publics on different platforms, a video channel where one member posts monologues that the others dissect, and a set of direct-message conversations the researcher will never see and should probably never want to. The methods literature offers two tempting responses to this situation, and both are wrong. The first is to treat the online setting as simply a new location for old methods. Ethnography moves to the forum; the interview happens over video; the fieldnotes get typed rather than scrawled. On this view nothing fundamental has changed, and the researcher's job is to apply Malinowski's injunction to grasp the native's point of view in a place that happens to have a URL. This response has the virtue of taking interpretation seriously. Its defect is that it underestimates how much the setting does to the data. A forum post is not an utterance overheard; it is a durable, searchable, platform-formatted, algorithmically ranked artefact composed for an audience the writer could only guess at, and possibly rewritten three times before posting. Reading it as though it were speech in a circle of chairs misses most of what is going on. The second response is to treat the online setting as a data source and reach for scale. Here the fourteen thousand members become four million posts, the posts become a matrix, the matrix gets a sentiment score and a topic model and a network layout, and the resulting patterns are reported as findings about how people who have been through this illness talk. This response has the virtue of taking the size of the archive seriously. Its defect is that the numbers it produces are often measurements of the platform rather than of the people. A spike in negative sentiment may record a change in who was posting, or a change in the ranking algorithm, or the arrival of a brigading campaign, or nothing more than the fact that a lexicon built on product reviews scores the word "positive" as good and so reads "my biopsy came back positive" as joy. This booklet argues for a third position, and it is not a compromise between the other two. The claim is this: qualitative inquiry online is distinguished by the work it does to keep interpretation attached to context, and that work has to be designed into the ethics, the access route, and the analytic method simultaneously, because each of those three decisions destroys or preserves a different part of the context the interpretation depends on. That is a single, demanding idea, and everything that follows is an unpacking of it. Consider what it rules out. It rules out doing the ethics at the end, as a compliance exercise, because the consent route you choose determines whether you can quote, which determines whether your reader can evaluate your reading. It rules out treating data acquisition as a technical preliminary, because whether you obtain material through an official research interface, a data donation, a public archive, or your own careful reading determines what metadata survives, what sampling frame exists, and what you are permitted to say about the population. It rules out bolting computational text analysis onto a qualitative study as a garnish, because a classifier trained on decontextualised strings and a close reading of a thread are not two views of the same object; they are answers to different questions, and the researcher has to know which question is being asked. Context is the word doing the heavy lifting here, so it is worth being precise about it. Helen Nissenbaum's account of privacy as contextual integrity gives us the sharpest available version: information flows carry norms attached to the context in which they occur, and a breach happens not when private information becomes public but when information moves from one context into another under different norms. A post in a recovery forum is not secret; it is contextually situated. Moving it into a journal article, or into a training corpus, or into a spreadsheet of sentiment scores, is a flow across contexts, and each such flow has to be justified rather than assumed. But context is also an epistemic matter, not only an ethical one. The meaning of a message depends on the thread it answers, the community's history of similar exchanges, the platform's affordances for sarcasm and quotation, the poster's standing in the group, and what everyone knows happened last week. Strip that away and you have a string of characters that can be counted but not understood. The characteristic error of weak digital qualitative work is not that it is too small or too impressionistic; it is that it takes decontextualised material and interprets it as though the context were still attached. What this booklet covers, and what it does not The chapters move roughly in the order a study moves. The first chapter takes on the question of what the field is when it has no location, and how researchers from Christine Hine onwards have answered it. This matters practically: your definition of the site is your sampling frame in disguise, and most weak designs are weak because the site was never defined at all. The second chapter puts ethics where it belongs, which is before method rather than after it. The public–private binary is useless online; contextual integrity, risk-based reasoning, and honest thinking about what a quotation can do to a person are what replace it. Chapters three and four cover the two classical field methods in their online forms: participant observation, where the difficulty is presence rather than access, and asynchronous interviewing, which is not a degraded version of a face-to-face interview but a genuinely different instrument with its own strengths. The fifth chapter addresses how a corpus is lawfully assembled. This is the least glamorous chapter and possibly the most useful one. It treats the API, the regulated research-access channel, the data donation, and the public archive as distinct instruments with distinct properties, and it is unapologetically clear that evading a platform's technical controls is not a method. Chapters six, seven, and eight cover analysis: close discourse analysis of online talk, computational text methods including sentiment analysis, and the mining of relational structure. The argument across those three chapters is that computational methods are best understood as ways of finding places to read closely, and that their outputs are hypotheses about a corpus rather than findings about people. The ninth chapter is about writing, which online research makes unusually hard because verbatim quotation is searchable and therefore identifying. Solutions exist; they involve trade-offs the reader of your work deserves to be told about. Several things are deliberately out of scope. This is not a book about how to circumvent platform controls, defeat rate limits, or re-identify individuals from de-identified data; where those topics arise they are treated as things researchers must not do and as risks that responsible design mitigates. It is not a statistics text, and it does not teach any particular software. It is not a survey of every platform, because platforms change faster than books are printed and a method tied to one interface has a shelf life measured in months. And it does not pretend that the four-year-old tooling landscape is stable: the closure of CrowdTangle in August 2024, the collapse of free academic access to what was then Twitter's API in 2023, and the arrival of a regulated European research-access regime under the Digital Services Act have all rearranged the practical terrain in ways this booklet describes but cannot freeze. Who this is for The reader I have in mind has done, or is about to do, empirical social research and has a reasonable grasp of what interpretation involves. They may be a doctoral student working out whether their project is feasible, a researcher in a policy or public-health setting who needs to understand online conversation without a computational team, or a computational social scientist who has begun to suspect that their models are answering a question nobody asked. What the booklet tries to give all three is judgement rather than procedure. There are procedures here — how to build a fieldnote practice when there is no drive home, how to structure an asynchronous interview so it does not die after the second exchange, how to validate a sentiment classifier against a hand-coded sample, how to decide whether a quotation is safe to print — but procedures without judgement produce studies that are correct in every step and worthless overall. The underlying stance is not sceptical about computation. Large-scale text and network methods have genuinely extended what qualitative researchers can see; a topic model that surfaces a register of talk you had not noticed in eleven thousand posts has done you a service no amount of close reading would have. The stance is sceptical about unvalidated computation, and about the particular illusion that the digital record is a transcript of social life rather than a trace of interaction with a commercial system that was designed for other purposes. Two commitments follow, and they run through every chapter. The first is that no quantitative output about text or ties is a finding until someone has read enough of the underlying material to say what it means. The second is that the ethical obligations of this work do not weaken as the data gets bigger. A research subject does not stop being a person because there are four million of them. That is the argument. The rest is the working out. Chapter 1: The Field Without a Location Bronisław Malinowski's famous instruction to the ethnographer was to pitch a tent in the village, learn the language, and stay long enough that the strange becomes ordinary. The instruction assumed something so obvious that he never had to say it: that there was a village, that it had edges, and that being inside those edges meant something. Almost every difficulty in online qualitative research traces back to the fact that this assumption no longer holds, and to the various unsatisfying ways researchers have tried to patch it. The patches matter because the definition of the field is not a philosophical preliminary. It is the sampling frame, the ethical perimeter, and the claim of the eventual write-up, all bundled together and usually left implicit. A study that never decided what its field was cannot say what its findings are about. When a reviewer asks "how do you know this is characteristic of the community rather than of the loudest twelve accounts?", they are asking a question about site definition that should have been answered before any data was collected. Four ways of drawing the boundary The literature has settled into roughly four framings, and they are not interchangeable. Each defines the field by a different principle, and each buys a different kind of validity at a different cost. The bounded-platform framing treats a particular forum, subreddit, server, or group as the site, in more or less the way the village was a site. This is what most early internet ethnography did, and it remains the right choice when the community really does have a membership, a shared history, and a moderating structure that constitutes it as a group. Tom Boellstorff's work in Coming of Age in Second Life (2008) is the most thoroughgoing defence of the position. He argued, against the reflex that virtual worlds are somehow less real, that a world with its own economy, norms, and social hierarchies can be studied in its own terms, without constant reference to the offline lives of the people playing. That is a methodological argument about boundaries: the field is where the action is, and the action can be inside the world. The networked-practice framing, associated above all with Christine Hine, denies that the site is a place at all. In Virtual Ethnography (2000) and more fully in Ethnography for the Internet (2015), Hine argued that the internet is at once a culture in its own right and a cultural artefact embedded in other settings, and that the ethnographer follows connections rather than staying put. The field becomes a trajectory: this forum, plus the Discord it spills into, plus the conference where members meet, plus the domestic contexts where people read it on their phones at midnight. George Marcus's older idea of multi-sited ethnography, of "following the thing" or "following the metaphor," is the obvious ancestor. The event or phenomenon framing defines the field by something happening rather than somewhere existing: a controversy, a hashtag surge, a moderation crisis, a product launch. Here the boundary is temporal and thematic. Much of what is published as social media research is implicitly of this kind, and it is a legitimate design provided the researcher acknowledges that they are studying an event's public trace and not a community. The person-centred framing puts a set of participants at the centre and treats the field as whatever media practices those people engage in. This is the design behind the UCL Why We Post project, published as Daniel Miller and colleagues' How the World Changed Social Media (2016), in which nine ethnographers spent fifteen months each in different locations and studied social media as one strand of local life rather than as a separate world. It is also the logic of most work that pairs interviews with participants' own accounts of their platform use. The finding such a design supports is about people; it cannot easily support claims about a platform in general. The choice among these is a choice about what you will be able to claim, as Table 1 sets out. Table 1. Four ways of defining the online field and what each supports. Framing Boundary principle Claims it supports Characteristic failure Bounded platform Membership in one forum or world Norms, roles, and internal culture of that group Treating one community's culture as "how people online behave" Networked practice Connections followed across settings How a practice travels and changes across contexts Drift; a field so diffuse that nothing is observed in depth Event or phenomenon A controversy or period of activity The shape and repertoire of a public episode Mistaking the trace of an event for a community Person-centred A set of participants and their media use How specific people integrate platforms into life Overreach from a small purposive sample to a platform None of these is more rigorous than the others. What is not rigorous is choosing none and letting the field be defined by whatever the search interface happened to return. The three things that make a digital site different Even once a boundary is drawn, the site has properties a physical one does not, and three of them bear directly on method. The first is persistence. Most of what is said online is written down and stays written down. This is an extraordinary gift and a trap. The gift is that the ethnographer can reconstruct a history they were not present for: the argument in 2019 that split the group is still readable, complete with the deleted-then-restored post everyone still talks about. The trap is that the archive tempts the researcher to substitute retrieval for participation. Reading five years of a forum in a week is not the same as having been there for five years, because you read it knowing how it turned out, in a compressed sitting, without the intervening ordinary days. The archive gives you the events and takes away the rhythm, and rhythm is much of what ethnography is for. The second is searchability and audience uncertainty. Because material is retrievable, its audience is not the audience present at the time of writing. danah boyd's account of networked publics identifies this as the structural condition of online sociality: the audience is invisible, contexts collapse, and public and private are not distinct states but a negotiated spectrum. For the researcher this has two consequences. Analytically, participants are writing for imagined audiences that include people not yet present, which shapes what they say in ways the text itself may not reveal; the interview is often the only way to find out who someone thought they were talking to. Ethically, the researcher is themselves one of those future audience members, and a quotation republished in a journal is a further collapse of context that the writer never consented to. The third is algorithmic mediation. What is visible on a platform is the output of a ranking system optimising for engagement, tuned continuously, and personalised to the viewer. Two researchers observing "the same" feed on the same day see different things. This breaks a basic ethnographic assumption — that what is in front of you is what is happening — and it has no clean fix. What it requires is documentation: record the account you observed from, its follow graph, its history, the date, and any known platform changes, and treat your observations as observations of one algorithmically constituted view rather than of the platform as such. Zeynep Tufekci's discussion of the "denominator problem" in social media research makes the general point sharply: without knowing the population from which visible material was drawn, proportions computed on it mean very little. Digital, virtual, networked: why the adjectives disagree Newcomers are often confused by the proliferation of names — virtual ethnography, netnography, digital ethnography, social media ethnography, online ethnography — and assume they are branding exercises. Mostly they are not; they encode different commitments. Virtual ethnography (Hine) came first and carried the early 2000s question of whether online interaction was real enough to study. Its answer was yes, and its method was to treat the internet as both culture and artefact. Netnography is Robert Kozinets's codified procedure, developed initially in consumer research and set out most fully in the successive editions of Netnography (2010; revised 2020). It is more prescriptive than most anthropological approaches — with defined stages of entrée, data collection, interpretation, and member checking — and it is candid about researching communities for applied and commercial purposes. That prescriptiveness is genuinely useful for students and for applied projects with timelines; it can also encourage a checklist relationship to fieldwork. Digital ethnography as used by Sarah Pink and colleagues in Digital Ethnography: Principles and Practice (2016) widens the frame again: the digital is not a separate site but part of everyday environments, studied alongside the sensory and material aspects of life. John Postill and Pink's earlier account of "social media ethnography" described the researcher's practice as catching up, sharing, exploring, interacting, and archiving — a messy set of daily routines rather than a residence. The practical upshot is not that one label is correct. It is that the labels come with different default answers to the boundary question, and choosing a label without choosing a boundary is how projects end up incoherent. Choosing a site you can actually study Three tests separate a workable site from an aspiration. Is there enough interaction to observe? A great many online spaces are archives of monologue: posts with no replies, comments with no responses to the comments. If your interest is in interaction — how members respond to a newcomer's disclosure, how a norm is enforced — then a site without threads cannot give it to you, however large it is. Before committing, read fifty randomly chosen items and count how many are parts of exchanges. This takes an afternoon and saves months. Is there a history you can reconstruct? Communities are constituted by their crises. If the platform deletes old content, or the group's history is in a channel you cannot access, you will be studying a permanent present, which makes it very difficult to tell a norm from a mood. Can you get lawful, documented access to it, at the depth the question requires? This is where many otherwise good designs die, and it is worth doing the feasibility check first rather than last. A question about private-group dynamics requires either participation with the group's knowledge or interviews; it cannot be answered from public data, and no amount of methodological ingenuity changes that. Chapter 5 deals with the access routes in detail. The point here is that site selection and access route are one decision, not two. A fourth consideration is less a test than a warning. Ease of access is not a reason to pick a site, and it is the single most common reason sites actually get picked. For roughly a decade, a disproportionate share of published social media research studied Twitter, for a reason that had nothing to do with Twitter's social importance: its API was open, its data was cheap, and its posts were short enough to model. Derek Ruths and Jürgen Pfeffer set out the resulting distortions in Science in 2014, and Tufekci made the related argument that Twitter had become the "model organism" of social media research — the fruit fly whose convenience shaped the questions the field asked. When academic access to that platform was withdrawn in 2023, a substantial body of accumulated method turned out to be tied to an interface rather than to a problem. The lesson is not to avoid popular platforms; it is to be able to say, in one sentence, why this site answers this question, and to have that sentence survive the platform changing its pricing. Where offline and online stop being separable An older debate asked whether online research needed offline "grounding" to be valid. That debate is over, and it ended in a draw that is worth understanding, because the draw is the working position most good projects now adopt. The position that online life is real life, studiable on its own terms, won the argument against dismissal. Nobody now seriously claims that a decade-long friendship maintained in a group chat is not a real friendship, or that norms enforced in a moderated forum are not real norms. But the position that online settings are separate worlds lost too, and lost harder. Nearly all the settings a contemporary researcher studies are continuous with the rest of participants' lives: the recovery forum's members go to clinics, the political community organises offline events, the fandom has jobs and rent. Miller's comparative work makes the case with unusual force, because it found that what social media did varied enormously by place — the same platforms functioned as conservative instruments of family surveillance in some settings and as instruments of escape in others. A study of the platform alone could not have seen that. The practical resolution is that the boundary of the field is set by the question, not by the technology. If the question is about how a moderation norm gets enforced, the norm's enforcement is observable in the forum and you may not need to leave it. If the question is about what membership in the group means to people, you will have to ask them, and what they tell you will be full of things you could not have seen. This has a consequence worth stating plainly, because it recurs in every subsequent chapter. Observation alone rarely answers questions about meaning. You can see what people do in a thread. You cannot see what they thought they were doing, who they thought was reading, what they decided not to post, or what happened in the direct messages that made the public exchange look the way it did. The invisible portion of online interaction is enormous, and it is systematically invisible — the most sensitive material moves to the most private channel. Designs that rely wholly on public traces are therefore designs that have accepted a specific, structured blindness. That can be a reasonable trade. It is not a reasonable thing to leave unmentioned. The platform is a participant, not a venue One more thing distinguishes a digital field, and it is easy to miss because it is everywhere: the site has an owner, and the owner is an actor in it. In a physical field, the built environment shapes interaction but does not intervene in it. Online, the environment writes rules, enforces them selectively, changes the interface without notice, ranks what members see, deletes accounts, and occasionally deletes the community. Any account of norms in an online group that ignores the platform's own governance has left out one of the three parties to every interaction — the poster, the audience, and the system that decides whether the post is seen. This shows up concretely in fieldwork. A sudden change in the tone of a forum may be a change in its membership, or it may be that a moderation tool was introduced. A drop in the visibility of a topic may reflect declining interest or a change in ranking. Members themselves theorise about this constantly — the vernacular vocabulary of "shadowbanning," "the algorithm," "getting reach," and deliberately misspelled words to evade automated detection is evidence that participants experience the platform as an agent with intentions. Those folk theories are excellent data in their own right, whether or not they are technically accurate, because they shape what people write and how they write it. There are two practical implications. The first is that a study should record the platform's governance context alongside its social one: the terms of service and community guidelines in force during the study period, notable policy or interface changes, and any moderation events the community discussed. Guidelines change often and are rarely archived by the platform, so save dated copies as you go. The second is that the researcher should resist inferring from absence. Content that is not visible may never have existed, may have been deleted by its author, may have been removed by moderators, or may simply not have been surfaced to the account you were watching from. These are analytically different, and only the community's own talk about removals usually tells you which occurred. Gabriella Coleman's ethnographies of free-software developers and of Anonymous are instructive here precisely because their subjects were unusually articulate about infrastructure. In communities that build and argue about the systems they inhabit, the platform's role is stated aloud. In most communities it is not, and the researcher has to learn to hear it anyway. What to write down before collecting anything A site definition that exists only in the researcher's head is a site definition that will drift. Write it down, in a page, before collection begins, and keep the page as a document to be revised with dates rather than replaced. It should state: what the field is, in one sentence, in terms a stranger would understand; what is inside it and what is adjacent but outside; the period covered and why those dates; the platforms and specific spaces included; how you will know if the boundary turns out to be wrong; and what you expect to be unable to see. That last item is the one people skip and the one reviewers notice. If the design is a following design in Hine's sense, the page instead records the rule for following — which connections you will pursue and which you will decline — because without such a rule, multi-sited research becomes tourism. None of this is bureaucracy. It is the difference between a study whose claims have a referent and a study that has collected a large amount of material about roughly a topic. The rest of this booklet assumes the page exists. The next chapter argues that it should be written alongside a second one, about harm, and that neither is finished until both are. Chapter 2: Ethics as Method, Not Paperwork In 2008 a research team released a dataset built from the Facebook profiles of an entire cohort of undergraduates at one American university. The data was anonymised by the standards of the day: names removed, the institution identified only as "an anonymous north-eastern American university." Within days, Michael Zimmer had established which university it was, using nothing more than the dataset's own documentation — the cohort size, the unusual range of majors, the study-abroad destinations. He published the analysis in Ethics and Information Technology in 2010 under a title that has since become a slogan: "But the data is already public." The case is worth keeping in mind for two reasons. The first is the obvious one about re-identification. The second is subtler and more important: nothing the researchers did was malicious, careless by the standards then current, or unapproved. They had ethics board clearance. The failure was a failure of method — of thinking about data collection, anonymisation, and release as separable technical steps rather than as one design problem with ethics running through it. That is the argument of this chapter. Ethics in online research is not a form to be filled in before the interesting work starts. It is a set of design decisions that determine what data you have, what context survives with it, what you can quote, and therefore what you can claim. Get it wrong early and you will find, at the writing stage, that you cannot publish the evidence for your best finding. Why "public" is not a permission The most persistent error in this field is the inference from publicly accessible to fair game. It is worth taking apart carefully, because it is not stupid — it is just wrong in a specific way. Accessibility is a property of a technical system. A post on an open forum can be retrieved by anyone with a browser. From this it does not follow that the poster consented to research use, that they understood the post's retrievability, or that republishing it causes no harm. Legal availability and ethical appropriateness are different questions, and the answer to the second cannot be read off the first. Nissenbaum's contextual integrity, introduced in Privacy in Context (2010), gives the better frame. Every context has norms governing the flow of information within it: what may be said, to whom, under what conditions it may be passed on. Privacy is preserved when flows respect those norms and violated when they do not — including when information that was already "public" moves into a context with different norms. A person who posts about relapse in a recovery forum has shared information in a context whose norm is that it stays among people who understand it. Quoting that post in a paper, which may then be indexed and surfaced to the person's employer via a search engine, is a flow across contexts. It may still be justifiable. It is not automatically permitted by the fact that the forum had no login. The Association of Internet Researchers has built its guidance on essentially this insight. The successive AoIR documents — the 2002 guidelines, Markham and Buchanan's 2012 revision, and Internet Research: Ethical Guidelines 3.0 (franzke, Bechmann, Zimmer and Ess, 2020) — decline to provide rules, on the deliberate ground that rules do not survive contact with new platforms. What they provide instead is a case-based, question-driven process: ask what the participants' reasonable expectations were, what harm could follow, how vulnerable the population is, and whether the research value justifies the flow. Reviewers in this field increasingly expect to see that reasoning shown, not a citation to a principle. A useful way to make the reasoning concrete is to place the material on a spectrum rather than in a box, as Table 2 does. Table 2. Settings on the accessibility–expectation spectrum, and what each implies. Setting Technically accessible to Members' likely expectation Re-identification risk Usual consent route Open forum, indexed by search engines Anyone Mixed; many assume obscurity High for verbatim quotes Notice to community; individual consent for quotes Large open platform, public posts Anyone Aware of visibility; rarely of research High if quoted; low if aggregated Aggregate freely; seek consent to quote Closed group requiring approval Approved members Confidentiality within group Very high Negotiated access with moderators plus individual consent Semi-public channel shared by link Anyone with the link Obscurity by default Very high Treat as private; consent required Direct or ephemeral messages Intended recipient Fully private Absolute Consent from all parties; usually solicited data only The table is a heuristic, not a licence. Two considerations override it. One is topic sensitivity: the same forum is a different ethical object when the thread is about kitchen renovation and when it is about suicidal ideation. The other is population vulnerability: minors, people in legal jeopardy, people whose identity is criminalised where they live, and people in acute distress warrant protection well beyond what the setting's accessibility suggests. Harm, specified "Risk of harm" is a phrase that invites hand-waving. It helps to name the mechanisms, because each has a different mitigation. Identification harm occurs when a research output lets someone be identified who did not expect to be. The main vector online is verbatim quotation: a distinctive sentence pasted into a search engine returns the original post, and from it the account, and from the account potentially a name, employer, and location. This is the single most under-appreciated risk in the field, and Chapter 9 deals with the response in detail. It is worth noticing now that it makes online research different from interview research in a way that cuts against normal practice — in interview studies, verbatim quotation is a mark of rigour; in studies of written online material, it can be a disclosure. Aggregation harm occurs when individually innocuous material is combined into a profile. A person's posting history across a decade, assembled into one file, reveals a great deal that no single post did. Researchers routinely assemble such files without thinking of them as dossiers. Group harm occurs when a community is characterised in ways that damage it collectively, even if no individual is identified. Studies of stigmatised communities frequently produce this, and it is not addressed by anonymisation at all. Naming the community is often the problem; several researchers now pseudonymise the forum as well as the members. Disruption harm occurs when the research alters the setting. An ethnographer who joins a small group and asks a lot of questions changes it; a researcher who posts a recruitment notice to a support forum imports a research frame into a space that people use to not be studied. Downstream harm occurs when data leaves the study. A dataset shared for replication, a corpus deposited in a repository, a set of quotes reproduced in a press release — each is a further flow. Against this list, the researcher's commonest reassurance — "I will anonymise" — is a response to one mechanism out of five. Consent when consent is impractical The hardest practical question is what to do when the population is too large, too transient, or too anonymous to consent in the conventional sense. There is no single answer, but there is a defensible ordering of options, and the discipline is to work down it rather than jumping to the bottom. Individual informed consent remains the default where interaction is involved: interviews, participation, anything solicited. It is also achievable more often than people assume for quotation. Contacting an account to ask permission to quote a specific post works reasonably often, particularly if the request is specific and shows the intended text. Community-level consent — negotiating with moderators or administrators, posting a notice, allowing an opt-out period — suits participant observation in bounded groups. It is weaker than individual consent because moderators do not speak for members, but it satisfies the norm that a community should not be studied in secret. It also tends to improve the research: members who know a researcher is present will often tell them things. Waiver with mitigation is appropriate for large-scale study of public material where contacting individuals is genuinely impossible and the topic is not sensitive. Here the obligation shifts entirely onto mitigation: no verbatim quotation of identifiable material, no republication of usernames, aggregate reporting, and a paraphrase protocol for illustrative material. Retrospective or member-checking approaches, in which findings are shown to community members before publication, do not substitute for consent but frequently catch errors of interpretation and unanticipated harms. What is not on the list is the reasoning that because consent is hard, it does not apply. The size of a dataset has no bearing on whether the people in it are people. There is a specific hard case worth naming: research on communities the researcher considers harmful — extremist movements, harassment networks, organised disinformation. Here the consent norms genuinely bend, because informing a group that they are being studied may endanger the researcher and will certainly change what is said. Covert observation of such groups has a long and respectable tradition in offline sociology. The bending is not unlimited, though: the researcher still owes non-identification to individuals who are not public figures, still owes accuracy, and still has to have submitted the design for review rather than deciding alone. Researcher safety planning belongs in the same conversation, since studying such groups carries a real risk of retaliation. Law, regulation, and terms of service Ethics and law are separate, and confusion between them causes trouble in both directions — researchers who assume that lawful means permissible, and researchers who assume that any terms-of-service question makes a project impossible. Data protection law applies to most of this work. Under the EU General Data Protection Regulation, online posts about identifiable people are personal data whether or not they are public, and some — health, political opinion, sexual orientation, religion — fall into the special categories that require additional justification. GDPR does contain provisions that accommodate research: Article 89 permits derogations for scientific research subject to safeguards, and a public-interest or legitimate-interests basis is often available where consent is not. The practical effect is not that research is exempt but that it must be documented — purpose, lawful basis, minimisation, retention period, security. UK and EU institutions will generally require a data protection impact assessment for a project of any scale. Note that the regulation applies to pseudonymised data, since pseudonymised data remains personal data; only genuinely anonymous data falls outside, and true anonymisation of text is very difficult. Human-subjects regulation in the United States takes a different route, through the Common Rule and institutional review boards. The definition of human-subjects research turns on interaction or identifiable private information, which leaves a substantial category of public online data outside formal IRB jurisdiction. This exemption is a jurisdictional fact, not an ethical verdict; the Menlo Report (2012), which adapted the Belmont principles for information and communications technology research, was written partly because so much consequential work fell outside the existing framework. Terms of service are the most misunderstood layer. Platform terms typically restrict automated collection, bulk storage, and redistribution. Breaching them is a contract matter, and the main practical consequences are account termination, loss of access, and institutional exposure; several journals and funders now ask directly whether data was collected in compliance. In United States law the question of whether breaching terms also constitutes computer misuse has narrowed considerably: Van Buren v. United States (2021) read the Computer Fraud and Abuse Act's "exceeds authorized access" clause narrowly, and the hiQ Labs v. LinkedIn litigation in the Ninth Circuit held that scraping publicly available data was unlikely to constitute unauthorised access under the statute — while leaving contract and other claims alive. In Sandvig v. Barr (2020) a district court held that mere terms-of-service violations in the course of research did not constitute a criminal CFAA offence. The net position is that the criminal risk to an academic researcher collecting public data is much lower than the folklore suggests, and the contractual, institutional, and ethical constraints are entirely intact. This booklet takes a firm line on what follows. Working within an interface a platform provides, on terms it publishes, is research. Defeating technical access controls, using credentials that are not yours, creating accounts to gain entry to private spaces under false pretences without review, or attempting to re-identify individuals in a de-identified dataset are not methods and are not treated as such here. The reason is not timidity. It is that each of those acts destroys the thing the research depends on — the possibility of saying honestly, in the write-up, how the material was obtained. The regulated alternative is worth knowing about, because it is new. The EU's Digital Services Act creates, in Article 40, a mechanism by which vetted researchers can apply through national Digital Services Coordinators for access to data from very large platforms for research on systemic risks. The Commission adopted the delegated act setting out the operational rules in July 2025, and the regime has been coming into effect since. It is narrower than its advocates hoped — it is oriented to systemic risk rather than to social research generally, and it is institutionally heavy — but it is the first legal route to platform data that does not depend on a company's goodwill. Three cases worth having in mind Abstract principles are easier to apply when attached to episodes that actually happened, and this field has a small canon of instructive failures. The emotional contagion experiment reported by Adam Kramer, Jamie Guillory and Jeffrey Hancock in the Proceedings of the National Academy of Sciences in 2014 manipulated the emotional valence of content shown in hundreds of thousands of Facebook news feeds and measured the effect on users' own posting. The science was modest; the reaction was not. The core objection was that experimental manipulation of mood without consent is not made acceptable by being routine product optimisation, and the case established in most researchers' minds a distinction between what a platform may do to its users commercially and what a researcher may do to them scientifically. The journal appended an editorial expression of concern. The durable lesson concerns collaboration: when academic researchers work with platform data scientists, the ethical regime that applies is the academic one, and it has to be established before the experiment, not defended after it. The OkCupid dataset released in 2016 by a researcher who posted profile data on tens of thousands of users to an open repository — including usernames, ages, locations, and answers to intimate questionnaire items — is the purest available illustration of the "already public" fallacy. The data had been accessible to logged-in users of the service. Compiling it into a downloadable file changed what it was, from scattered profiles readable one at a time by people with a reason to look, into a searchable dossier. The repository removed it. No anonymisation was attempted, and the defence offered was that none was needed because nothing had been hidden. The hacked-data question arises whenever a breach dumps a corpus that would answer a research question: the Ashley Madison leak in 2015 is the standard example, and there have been many since. The material is undeniably available and undeniably informative. The arguments against using it are that the data subjects suffered a harm the researcher would be extending, that the research would create an incentive structure rewarding breaches, and that provenance cannot be verified. The narrow exceptions people defend — research on the breach itself, or on named public actors where the public interest is substantial — prove the rule by how narrow they are. The general position is that a dataset's origin is part of its ethics, and that "someone else did the wrong" is not a laundering mechanism. What the three have in common is that each involved competent researchers who had asked whether the data was available rather than whether the flow was appropriate. Building the ethics into the design Practically, this chapter's argument reduces to doing four things at outline stage rather than at submission stage. Decide the consent route before choosing the site, because the route constrains which sites are feasible. Decide the quotation policy before collecting, because it determines whether you need to be able to contact posters. Decide the retention and storage plan before the first file exists, because retrofitting security to a folder of scraped posts on a laptop is not really possible. And write down, in advance, what you will do if you encounter disclosure of imminent harm — this happens more often in online fieldwork than researchers expect, and deciding in the moment goes badly. A practice that helps with all four is to keep an ethics log beside the fieldnotes: a dated record of decisions, the reasoning behind them, and the things that unsettled you. It costs ten minutes a week. It produces, at the end, both the material for an honest methods section and the evidence that the judgement calls were judgement calls rather than defaults. Chapter 3: Being There When There Is No There The classical justification for participant observation is that some things can only be learned by being present while they happen. You cannot get the texture of a workplace from interviews alone, because people describe their jobs in terms of official purposes and forget the improvisations that actually make the place run. The ethnographer's presence over time produces knowledge of a specific kind: tacit, comparative, and grounded in having seen enough ordinary instances to recognise an unusual one. Online, the difficulty is not access — you can read the forum this afternoon — but presence. Access is trivially available and presence is genuinely hard, which is the reverse of the offline situation and the reason that much published "online ethnography" is not ethnography at all but a content analysis with a longer reading period. This chapter is about how presence is constructed when there is nowhere to stand. The lurking question Every online fieldworker begins as a reader and has to decide how long to stay one. Reading without posting is not a moral failing. It is, in most online spaces, the majority practice; the great bulk of any community's membership never posts, and the researcher who only reads is behaving like most members. In spaces where reading is the normal mode, a period of sustained reading is genuine immersion rather than a substitute for it. But reading without posting has two specific limits. First, it does not test understanding. The ethnographer who participates finds out when they have got something wrong, because their contribution lands badly; the ethnographer who only reads can hold an incorrect interpretation indefinitely with nothing to correct it. Second, undisclosed reading forecloses the relationships that yield interviews, clarifications, and the eventual member checks that protect against misreading. There is also an ethical dimension. Observing a group at length without telling them is covert research, whatever the platform's accessibility. In an open, unbounded, high-traffic space this is usually unproblematic, since there is no group to inform. In a small, bounded community with a membership that would recognise a stranger's presence, it is a decision that needs justifying rather than a default. A workable default: read widely and without announcement while scoping, because scoping is not yet the study; disclose before the study proper begins if the community is bounded enough for disclosure to mean something; and, once disclosed, participate as a member would, at whatever level is honest for someone in your position. What that level is varies enormously, and Table 3 is not needed to say so — the honest description is that you should be a recognisable kind of participant rather than a category-defying one. Disclosure as a practical problem Telling a community you are studying it is easy to say and awkward to do. Three approaches recur. The moderator route is the most common for bounded groups: approach the moderators or administrators privately, explain the project, ask permission, and agree what the membership will be told. Its advantage is that moderators know their community and will tell you things about it in the negotiation itself — often the single most informative conversation of the early fieldwork. Its risk is capture: a study conducted with the moderators' blessing can quietly become a study conducted from the moderators' point of view, and moderators are usually in conflict with some part of the membership. The profile route works in spaces with persistent identity: the researcher's account states plainly who they are and what they are doing, and their posts are visible as coming from someone doing research. This is passive disclosure. It works where profiles are read and fails where they are not. The periodic announcement route suits long studies in high-turnover spaces: a post at intervals restating the project, with an email address and an opt-out. It is the only approach that deals with the fact that a community's membership in month twelve is not the membership that was informed in month one. None of these achieves informed consent from everyone observed, and the honest methods section says so. What they achieve is that the research is not secret, which is a lower but real standard. Expect disclosure to change the setting, and treat the change as data. Communities respond to being studied in patterned ways — a flurry of meta-discussion, some hostility, some enthusiastic volunteering, some members going quiet. The hostility is often the most informative: what people fear a researcher will do with their words tells you about the community's experience of exposure, which is frequently the very topic under study. Constructing presence Presence is made of three things that come free offline and have to be built online: rhythm, co-presence, and accountability. Rhythm means being there on the community's schedule rather than your own. Every online space has a temporal structure — the forum where substantive posts appear on weekday evenings, the chat server that is dead until a European afternoon, the group whose real activity happens in a two-hour window after a weekly broadcast. A researcher who reads in one long weekly session experiences the space as an archive. A researcher who checks in daily at the right times experiences it as a place where things are happening now, with the attendant sense of waiting, missing things, and catching up. Postill and Pink's description of social media ethnography as a set of daily routines — catching up, sharing, exploring, interacting, archiving — is a description of rhythm construction. Practically this means setting a schedule and keeping it for the duration, including through the dull weeks. The dull weeks are where the baseline comes from. Co-presence means being in the space at the same time as others, which written asynchronous platforms make oddly difficult. The methods that generate it are live ones: voice channels, streams, scheduled events, synchronous chat, games. Boellstorff's virtual-worlds fieldwork was possible partly because avatars produce a strong sense of shared presence; text forums produce almost none. Where a community has any synchronous component, it usually repays disproportionate attention, because it is where the relationships that sustain the asynchronous parts are made. Accountability means being someone the community can hold responsible. It is produced by the small, cumulative business of using a consistent identity, answering when addressed, acknowledging correction, following the norms including the trivial ones, and not disappearing. It is also produced by restraint: not asking too many questions too early, not treating every exchange as an elicitation, not being the member who only ever appears when something interesting happens. There is a version of online ethnography that consists of joining forty spaces and being present in none. It produces breadth of a kind. It does not produce the thing participant observation is for. Fieldnotes without the drive home Offline ethnographers have a structural advantage they rarely notice: the journey home. The interval between the field and the desk is when raw experience gets turned into something writable. Online there is no interval. The field is the same screen as the notes, and the temptation is to skip the notes entirely because the material is already saved. That temptation should be resisted absolutely, because saved material is not fieldnotes. Robert Emerson, Rachel Fretz and Linda Shaw's Writing Ethnographic Fieldnotes (1995) makes the point that fieldnotes are already interpretation — selection, description, and the first analytic moves — and that the writing is where the ethnography happens. An archive of posts contains none of that. It contains what was said, not what it was like, what you noticed, what surprised you, or what you failed to understand. A workable online fieldnote practice has four layers, kept separate. The log is a dated record of what you did: which spaces you visited, for how long, what you posted, what you collected. It is dull and it is what makes the eventual methods section true. The observational note is written immediately after each session, in prose, describing what happened as though to someone who was not there. The discipline is to describe before interpreting: who spoke, what the exchange was about, how it went, what the tone was. Twenty minutes of this after a session is the core of the practice. The analytic memo is where interpretation is allowed. Written less often — weekly is usually right — it develops emerging ideas, notes patterns, and, crucially, records when a previous interpretation has been overturned. The overturning is the evidence that the fieldwork is doing something. The reflexive note records the researcher's own position and reactions: irritation at a member, boredom, the pull of taking sides in a conflict, the feeling of being an intruder. This is not self-indulgence. In online fieldwork particularly, the researcher's reactions are the main instrument for detecting the emotional texture of a space, and unexamined reactions leak into analysis as unexplained judgement. Screenshots and saved threads attach to the observational note as exhibits, with the date and the account they were viewed from. Save the context, not just the item: a reply with the post it replies to, a comment with the thread around it. Decontextualised captures are almost impossible to interpret six months later, and six months later is when you will be writing. One specific discipline is worth adopting: note what you did not understand. Online spaces are dense with in-jokes, abbreviations, references to prior events, and deliberately oblique phrasing. The impulse is to look up the unfamiliar and move on. The better practice is to record the incomprehension, because the list of things that were once opaque and later became obvious is a precise record of what you learned, and it is the material from which you will later explain the community to readers who start where you started. Participation without distortion The reflexive worry of all participant observation — that the observer changes what is observed — takes a particular form online, because contributions are permanent, public, and quotable. An offline ethnographer's remark in a meeting is gone in a moment. A researcher's post in a forum is a durable artefact that will be read by people who join later, may be quoted back, may be screenshotted into another context, and constitutes a standing position on whatever it addressed. The asymmetry argues for a more conservative participation style than one would adopt offline: contribute genuinely, but be aware that every contribution is also a publication. Two specific traps are worth naming. The first is elicitation by post. Asking the group a question is fast and produces material, and it is very tempting. But a question posted by a known researcher reframes the space as a research site for as long as the thread lives, and it produces answers of a particular kind — considered, performed, addressed to the researcher and to the audience simultaneously. Used sparingly and with awareness, this is a legitimate method. Used as the main mode, it turns ethnography into a slow, public, badly controlled survey. The second is the standing that comes from attention. Researchers are attentive, articulate, interested in what people say, and generous with replies. In many online communities this is rare, and it earns status quickly. A researcher can find themselves, within months, an influential member of the community they are supposed to be observing. This is not always avoidable and is not always bad — it produces access — but it needs recording in the reflexive notes and reporting in the write-up. Following across platforms Few communities live in one place any more. A group with a public forum will have a chat server, some members will have a shared channel on a messaging app, and a subset will be connected on a general-purpose platform where they also conduct the rest of their lives. The fragmentation is not incidental; it is how people manage audience. Material that is fit for the public forum goes there, material for the inner group goes to the chat, and the genuinely sensitive goes to direct messages. This produces the practical problem of how far to follow. The maximal answer — follow everywhere the members go — is unworkable and often unethical, since consent obtained in one space does not extend to another and since the more private spaces are private for reasons. The minimal answer — stay in the one space you negotiated access to — risks studying the community's front stage and reporting it as the whole performance, in the way Goffman's distinction between front and back regions in The Presentation of Self in Everyday Life (1959) would predict. The defensible middle is to follow deliberately and to document the rule. Decide in advance which spaces are in scope; negotiate access to each separately rather than assuming transitivity; and when a space is out of scope, say so in the write-up and say what you therefore cannot claim. If members tell you about what happens in the private channel, that is reported speech and can be treated as interview material, with the usual caveats about how people describe their own backstage. There is a related temptation to resist: assembling profiles of individual members by following them across platforms. Even where every element is public, the compiled picture is a dossier, and building one about a research participant without their knowledge is closer to investigation than to ethnography. Where biographical depth on particular members is genuinely needed, the route is to ask them. Leaving the field Online fieldwork has no natural end. There is no flight home, no last day, no ceremony. Studies therefore tend to trail off, with the researcher gradually reading less until they stop, which is both analytically sloppy and, in a community where relationships were formed, a small betrayal. Set an end date and observe it. Announce departure in the spaces where you disclosed presence, thank people, say what will happen next and roughly when, and give a route by which members can reach you about the eventual output. If you promised member checking, do it before the announcement rather than after, while people still recognise your name. Decide also what happens to your account and your contributions. Deleting a research account removes your own posts from threads and can make other people's contributions unintelligible, so deletion is rarely the right answer; leaving the account dormant with a profile note explaining the project's status usually is. If the community's norms expect departing members to say goodbye, say goodbye in the form the community uses, not in the form an institution would draft. Two further obligations tend to be forgotten. The first is to the moderators who granted access: send them the eventual output, or a readable summary of it, before it appears rather than after. The second is to yourself, in the form of a final reflexive memo written at the point of exit, describing how the community looked to you at the end and how that differed from the beginning. Written a year later, when the analysis is finished, that memo will be the only surviving record of what you actually believed while you were in the field, as opposed to what you concluded afterwards. Knowing when observation has run out Participant observation answers questions about practice: what people do, how interaction is organised, what the norms are and how they are enforced, what counts as a competent contribution, how newcomers are inducted, how conflicts unfold. It does not answer questions about meaning, motivation, or the unobserved. It cannot tell you why someone posted, what they considered and discarded, what the private channel said, how the group figures in the rest of their life, or what they think the researcher wants to hear. Online, the unobserved portion is unusually large, because the platform architecture reliably sends the most consequential talk to the least visible channel. The practical consequence is that virtually every serious online ethnography is a mixed design, in which observation establishes what happens and interviews establish what it means to the people it happens to. Kozinets's netnographic procedure formalises this, and most anthropologically inclined work arrives at it informally. Knowing when to move is a matter of judgement, but there is a reliable signal: you have been observing long enough when you can predict, with reasonable accuracy, how the community will respond to a new event — a newcomer's first post, a policy change, a member's departure — and when the predictions start being right. At that point further observation yields diminishing returns, and the questions that remain are questions only participants can answer. The next chapter is about how to ask them, in a medium where the conversation may take three weeks. Hashtags: #QualitativeInquiryInDigitalSpace #DigitalEthnography #OnlineEthnography #Netnography #VirtualEthnography #DigitalFieldwork #ParticipantObservation #AsynchronousInterviewing #OnlineDiscourseAnalysis #ComputationalTextAnalysis #SentimentAnalysis #TopicModeling #NetworkMining #SocialNetworkAnalysis #ContextualIntegrity #DigitalResearchEthics #OnlineConsent #ReIdentificationRisk #PlatformGovernance #AlgorithmicMediation #DataDonation #PlatformAPIs #DigitalServicesAct #MixedMethodsDigitalResearch #FutureOfDigitalQualitativeInquiry

Download the Book (PDF): Introduction A proteomics experiment never measures proteins. It measures ions: charged fragments of peptides, which are themselves fragments of proteins, flying through electric fields in an instrument that sees only mass-to-charge ratios and arrival times. Everything a biologist eventually reads in a results table, whether a list of eight thousand quantified proteins, a volcano plot of changes after drug treatment, or a claim that a kinase is phosphorylated at a particular serine, is the output of a long chain of inference that starts with those ions and works backward to biology. Each link in the chain was chosen by someone. Most of the choices were made before the first sample was ever loaded. That is the argument of this book. Quantitative proteomics is a design discipline before it is a measurement discipline. The depth of a proteome, the precision of a fold change, the trustworthiness of an identification and the honesty of a p-value are not properties of the mass spectrometer. They are properties of an entire workflow, from the buffer used to lyse the cells to the statistical model that tests for differential abundance. The two acquisition strategies that dominate contemporary discovery work, isobaric labelling with tandem mass tags (TMT) and data-independent acquisition (DIA), are best understood not as rival technologies but as two coherent sets of design decisions, each of which buys certain strengths by accepting certain costs. Choosing between them well requires understanding where those costs come from. The problem proteomics is trying to solve The human genome encodes roughly twenty thousand proteins. That number is deceptively small. Alternative splicing, proteolytic processing and hundreds of kinds of post-translational modification multiply it into a vast population of distinct molecular species, usually called proteoforms. More troublesome than the number is the range of abundances. In a typical cultured human cell line the most abundant proteins, such as histones, ribosomal proteins, cytoskeletal proteins and glycolytic enzymes, are present at millions of copies per cell, while transcription factors and many signalling proteins may be present at a few hundred or a few thousand. In blood plasma the problem is far worse: a handful of proteins, led by albumin, make up most of the protein mass, while the cytokines and tissue leakage markers of greatest clinical interest sit many orders of magnitude lower. Leigh Anderson and Norman Anderson's widely cited survey of the plasma proteome described that span as exceeding ten orders of magnitude. No analytical instrument covers such a range in a single measurement. A mass spectrometer has a finite capacity for ions at any instant, a finite number of spectra it can record per second, and a detection limit below which a signal disappears into noise. Every proteomics workflow is therefore an exercise in allocating scarce resources: separation capacity, instrument time, ion injection time and, eventually, statistical power. The practical question a researcher faces is never "how do I measure the proteome?" but "given what I want to know, how should I spend what I have?" Why bottom-up This book concerns bottom-up proteomics, sometimes called shotgun proteomics, in which proteins are enzymatically digested into peptides before analysis. The alternative, top-down proteomics, analyses intact proteins and preserves information about which modifications occur together on the same molecule. Top-down methods have advanced considerably, but for deep, quantitative interrogation of complex proteomes the bottom-up approach remains dominant, for reasons that are chemical rather than fashionable. Peptides of roughly seven to thirty amino acids are soluble, separate well by reversed-phase chromatography, ionise efficiently by electrospray, and fragment predictably in the gas phase. Their masses fall in a range where instruments are fast and sensitive. Proteins, by contrast, are heterogeneous in size, solubility and charge, and large ones fragment poorly. The price of digestion is the loss of the direct link between a measured peptide and the protein it came from. A peptide shared between two related proteins, or two isoforms of the same gene, cannot by itself say which of them was present. Reconstructing proteins from peptides, a problem known as protein inference, is one of the recurring difficulties the later chapters return to, because it affects identification, quantification and the calculation of error rates alike. Two philosophies of acquisition For most of the history of the field, discovery proteomics relied on data-dependent acquisition (DDA). The instrument surveys the peptides eluting from the chromatography column at a given moment, picks the most intense few, isolates each in turn, fragments it and records the fragment spectrum. The approach is intuitive and produces clean spectra that link one precursor to one set of fragments. Its weakness is that the choice of which peptides to fragment is partly random, driven by intensity and timing, so that the same peptide may be picked in one run and missed in the next. Across a study of many samples the result is a patchwork of missing values. The two strategies at the centre of this book represent two different answers to that weakness. Isobaric labelling, commercialised as iTRAQ and TMT in the early 2000s, attaches chemical tags to the peptides of several samples, pools them and analyses them together. Because the tags have identical total mass, the same peptide from every sample appears as a single precursor; only upon fragmentation do the tags release small reporter ions of distinct masses whose intensities reveal the relative amounts in each sample. If a peptide is picked for fragmentation, it is quantified in all samples at once. Missing values within a multiplexed set largely disappear. The costs are reagent expense, a ceiling on the number of samples per set, and a distortion called ratio compression, in which co-isolated peptides contaminate the reporter signal and pull measured ratios toward one. Data-independent acquisition takes the opposite route. Instead of choosing peptides, the instrument systematically fragments everything within a series of predefined mass windows, cycling through them fast enough to sample every eluting peptide several times across its chromatographic peak. The result is a complete, reproducible record of the fragment ions produced by every sample, at the cost of spectra that are highly multiplexed and therefore hard to interpret. DIA moved from a niche technique to a mainstream one in the decade after the SWATH-MS method was published in 2012, as faster instruments and new software, much of it based on machine learning, learned to untangle those spectra. Neither approach is universally superior. Controlled comparisons, including a 2019 study from Biognosys and the Leibniz Institute on Aging that matched instrument time between a TMT SPS-MS3 workflow and a single-shot DIA workflow, have found the familiar pattern: isobaric labelling identified somewhat more proteins and gave slightly better precision in that design, while DIA gave better accuracy of fold changes, and both detected true changes at similar rates. The balance shifts with sample number, sample amount, instrument generation, the biological question and the budget. A researcher who understands why each method behaves as it does can make that decision deliberately rather than by habit or by what the local core facility happens to run. Why error rates deserve a whole chapter The final third of the workflow, the bioinformatics, is where the most consequential and least visible decisions are made. A modern instrument can record tens of thousands of spectra per hour. Matching them to peptide sequences is a search problem with an enormous space of candidates, and some fraction of the matches will be wrong. The field's solution, the target-decoy strategy for estimating the false discovery rate (FDR), is one of the genuine intellectual achievements of computational proteomics, and it is widely misunderstood. A one percent FDR at the level of peptide-spectrum matches does not imply a one percent FDR at the level of proteins; error rates that are controlled in each run separately compound when the runs are merged; and a 2025 study using so-called entrapment experiments reported that none of the DIA search tools it examined consistently controlled the peptide-level FDR, with particularly poor performance on single-cell datasets. These are not esoteric concerns. They determine whether a reported protein list can be trusted. How the book is organised The chapters follow the path of a sample. Chapter 1 begins with sample preparation and digestion, the step where more experiments fail than at any other and where the bottom-up bargain is struck. Chapter 2 treats separation, both the online liquid chromatography that feeds the instrument and the offline fractionation that multiplies depth, as a resource to be budgeted. Chapter 3 explains what happens inside the mass spectrometer, how data-dependent acquisition works and why its stochastic sampling shaped everything that came after. Chapter 4 is devoted to isobaric labelling: the chemistry of TMT, the problem of ratio compression and the instrumental remedies for it, and the design of multi-batch studies. Chapter 5 does the same for DIA: window schemes, spectral libraries and library-free analysis, ion mobility, and the newest narrow-window methods. Chapter 6 turns to identification itself, from classical database search to predicted spectra and machine-learned rescoring. Chapter 7 is about counting errors, the logic of false discovery rates at every level from spectrum to protein. Chapter 8 follows the numbers from peptides to biological conclusions: normalisation, missing values, statistical testing and the question of how to choose between TMT and DIA for a given study. The conclusion argues what these threads imply for how proteomics experiments should be planned and judged. The reader assumed throughout is a working scientist from an adjacent field: a cell biologist who sends samples to a core facility, a geneticist who wants to integrate protein data with transcriptomes, a statistician asked to analyse a proteomics dataset, or a clinician evaluating a biomarker paper. No prior training in mass spectrometry is required, but the book does not simplify the substance. Where the field has settled a question, it says so. Where the question is open, it says that too. Chapter 1: From Tissue to Peptides The most expensive mass spectrometer in the world cannot recover a protein that was never extracted, a peptide that was never cut, or a sample that was contaminated with polyethylene glycol from a plastic tube. Experienced proteomics facilities tell a consistent story about failed projects: the cause is rarely the instrument and rarely the software. It is usually the sample. Sample preparation is where the bottom-up bargain is struck, where the proteome is converted into a peptide mixture whose composition will constrain every downstream measurement. It deserves to be treated as part of the experimental design rather than as a bench chore that precedes it. The goal of sample preparation can be stated simply. Starting from cells, tissue, biofluid or a purified complex, it should produce a clean mixture of peptides that represents the original proteins as completely and as reproducibly as possible, in a solvent compatible with reversed-phase chromatography and electrospray ionisation, and free of anything that suppresses ionisation or fouls the instrument. Every clause of that sentence hides a trade-off. Extraction, denaturation and the chemistry of artefacts Proteins must first be released from their biological matrix and unfolded so that a protease can reach its cleavage sites. The most effective way to solubilise membrane proteins, chromatin-bound proteins and aggregates is to use strong ionic detergents, above all sodium dodecyl sulfate (SDS), often combined with heat and mechanical disruption such as sonication or bead beating. SDS is also poison for mass spectrometry. It binds to reversed-phase columns, suppresses ionisation of co-eluting peptides and produces intense background ions. Chaotropes such as urea and guanidinium chloride denature proteins without the same chromatographic problems but are less effective at dissolving membranes, and urea brings a chemical hazard of its own: in solution it slowly decomposes to cyanate, which reacts with amine groups on proteins to form carbamylation artefacts. A urea buffer left warm overnight can modify a substantial fraction of lysine residues and peptide amino termini, shifting their masses and, in isobaric labelling workflows, blocking the very amines that the tags must react with. Much of the methodological history of proteomic sample preparation since 2009 is a history of ways to use detergents for extraction and then remove them before analysis. Three families of method now dominate, and each is worth understanding because each carries characteristic strengths. Filter-aided sample preparation, published by Jacek Wiśniewski, Matthias Mann and colleagues in Nature Methods in 2009, lyses the sample in SDS and then loads it onto a centrifugal ultrafiltration unit. The detergent is exchanged away by repeated washes with urea, the proteins remain on the filter, and digestion takes place on the membrane, after which the peptides are spun through. The method made SDS lysis compatible with shotgun proteomics and was widely adopted, though it is slow, involves many centrifugation steps and can lose material on the filter, especially at low input. Suspension trapping, commercialised as S-Trap and first described by Alexandre Zougman and colleagues in 2014, also begins with SDS lysis. Adding phosphoric acid and a high concentration of methanol causes the proteins to form a fine particulate suspension, which is trapped in a quartz or silica filter plug in a pipette tip or plate; the SDS is washed through, and digestion takes place in the trap. It is fast and handles large and difficult samples well. Single-pot, solid-phase-enhanced sample preparation, or SP3, introduced by Christopher Hughes, Jeroen Krijgsveld and colleagues in 2014 and refined in a 2019 protocol, uses carboxylate-coated paramagnetic beads. In the presence of a high concentration of organic solvent such as ethanol or acetonitrile, proteins aggregate onto the bead surface regardless of detergent. The beads are held by a magnet, the contaminants are washed away, and digestion proceeds with the proteins still associated with the beads. Later work showed the mechanism is better described as solvent-induced protein aggregation captured by the beads than as specific binding, which explains why the method tolerates almost any lysis buffer. Because SP3 works in a single tube or well and needs no centrifugation, it is readily automated on liquid-handling robots and scales down to low microgram and sub-microgram inputs, which is why it has become a default in many laboratories running large sample series. No one method is best for every sample. For formalin-fixed paraffin-embedded tissue, for example, the cross-links introduced by fixation must be reversed by prolonged heating before any of these methods can recover proteins efficiently; for plasma, the challenge is not extraction at all but dynamic range. What matters for design is consistency. Within a single study every sample should go through the same protocol, ideally in the same batch or in batches balanced across experimental groups, because differences in extraction efficiency between batches will otherwise masquerade as biology. Before digestion, disulfide bonds between cysteine residues are broken with a reducing agent, commonly dithiothreitol or tris(2-carboxyethyl)phosphine, and the freed thiols are capped with an alkylating agent so that they cannot re-form bonds. Iodoacetamide is the classical choice; it adds a carbamidomethyl group to cysteine, increasing its mass by about 57 daltons, and search software is told to expect that modification on every cysteine. Chloroacetamide reacts more slowly and produces fewer side reactions at non-cysteine residues, which is why several groups adopted it, but it was subsequently found to increase oxidation of methionine. These details sound minor. They are not. Every unexpected chemical modification introduced during preparation splits the signal of a peptide into several species of different mass, only one of which the search engine is looking for, and so reduces both identification rates and quantitative precision. Over-alkylation of lysines or peptide N-termini with iodoacetamide can even mimic the mass of biological modifications: a double carbamidomethylation of lysine has almost the same mass as the diglycine remnant that ubiquitination leaves behind after tryptic digestion, a coincidence that has caused false ubiquitination site assignments. The broader lesson is that sample preparation chemistry writes directly into the data. A search engine can only find what it is told to look for, and the unexplained spectra in any dataset are partly the fingerprints of chemistry that nobody accounted for. Digestion and clean-up Trypsin is the workhorse enzyme of bottom-up proteomics, and the reasons are instructive. It cleaves on the carboxyl side of lysine and arginine, residues that occur often enough in proteins that most tryptic peptides fall in a length range well suited to chromatography and fragmentation. Each tryptic peptide ends in a basic residue, which carries a positive charge; combined with the free amino terminus, this gives most tryptic peptides a charge of two or three in electrospray, and the basic C-terminus directs fragmentation to produce a clean series of y-type fragment ions that is easy to interpret. Trypsin is cheap, highly specific and active in the presence of moderate concentrations of denaturants. The textbook rule adds that trypsin does not cleave when the lysine or arginine is followed by proline. In practice, large-scale data have shown that cleavage before proline does occur at a meaningful rate, and modern search engines generally ignore the proline rule. More consequential are missed cleavages. Trypsin cleaves inefficiently at sites next to acidic residues, at sites with consecutive basic residues, and at lysines in structured regions. Missed cleavages generate longer peptides containing an internal lysine or arginine, splitting the signal of a region of the protein among several peptide species. A common remedy is to digest first with endoproteinase Lys-C, which cleaves after lysine, tolerates high urea concentrations and is more complete at lysine sites, and then with trypsin. Search settings typically allow up to two missed cleavages; a sudden rise in the missed-cleavage rate across a batch is one of the most useful quality-control signals that digestion went wrong. Trypsin also has blind spots. Regions of a protein that are very rich in lysine and arginine produce peptides too short to be informative, while long stretches without them produce peptides too long to detect well. Sequence coverage of a protein from a tryptic digest is therefore rarely complete, often well below half, and certain modification sites are systematically invisible. Alternative proteases, including chymotrypsin, Glu-C, Asp-N, Lys-N and alpha-lytic protease, produce overlapping but different peptide sets, and combining several digests raises coverage substantially. That is valuable when the goal is to map every modification site on a single protein or to identify sequence variants, but it multiplies instrument time, so for deep quantitative profiling of many samples trypsin alone remains the standard. Digestion is also where quantitative bias enters. Peptides are the unit of measurement, but the efficiency with which each peptide is released from its protein depends on local sequence and structure. That efficiency is constant from sample to sample only if the digestion conditions are constant. When they are not, peptide-level differences appear that have nothing to do with protein abundance, and they are indistinguishable from real changes unless the study design balances them. After digestion, peptides are desalted, usually on a reversed-phase C18 material, to remove salts, residual reagents and buffer components. The stop-and-go extraction tip, or StageTip, described by Juri Rappsilber, Yasushi Ishihama and Matthias Mann in 2003, packs small discs of C18 membrane into pipette tips; it is cheap, easily parallelised and still widely used. Plate-based cartridges and automated systems do the same job at scale. Peptide yield should then be measured, because loading a consistent amount of material onto the column is a precondition for reproducible quantification, and because the amount of peptide available sets hard limits on later choices such as how many fractions can be made and whether isobaric labelling, which requires a certain amount of material per channel, is practical. Contamination deserves specific mention because its effects are so disproportionate. Polymers such as polyethylene glycol, from detergents like Triton X-100 or from low-quality plastics and some hand creams, ionise extremely well and produce characteristic ladders of peaks spaced by 44 daltons that can dominate a spectrum and suppress peptide signals. Keratins from skin and hair are ubiquitous and are routinely included in search databases as contaminants so that they are recognised rather than misassigned. Residual SDS, even at trace levels, degrades chromatography. The trypsin itself autolyses and contributes peptides. None of this is exotic, and all of it is preventable with disciplined practice, but a single contaminated sample in a multiplexed set can compromise every sample it is pooled with. The special cases: plasma and low input Two kinds of sample push these principles to their limits and are worth treating separately because they shape so many current studies. Blood plasma and serum are the most clinically important and the most analytically hostile specimens in proteomics. Albumin alone makes up roughly half of the protein mass, and together with a small number of other high-abundance proteins such as immunoglobulins, transferrin, haptoglobin, fibrinogen and apolipoproteins, it accounts for most of the total. A shotgun analysis of neat plasma spends most of its instrument time on these proteins and typically quantifies several hundred proteins, not thousands. Three broad strategies extend depth. Immunoaffinity depletion columns remove the most abundant proteins before digestion, at the cost of expense, throughput, and the risk of co-depleting proteins bound to albumin or antibodies. Enrichment strategies, including perchloric acid precipitation and the use of engineered nanoparticles whose surfaces adsorb a protein corona that is enriched for lower-abundance species, compress the dynamic range in a different way; the nanoparticle approach, described by John Blume and colleagues in Nature Communications in 2020, has since been commercialised and used in large cohort studies. Finally, extensive fractionation, discussed in the next chapter, trades throughput for depth. Each strategy changes which proteins are measured and how reproducibly, so none of them can be treated as a transparent preprocessing step. A biomarker discovered after one kind of enrichment must be validated in a form that does not depend on it. Pre-analytical variation matters as much as analytical method in plasma work. The time a blood sample sits before centrifugation, the anticoagulant used, haemolysis, platelet contamination and freeze-thaw history all change the measured proteome. Philipp Geyer, Matthias Mann and colleagues catalogued proteins that act as markers of such problems, for example erythrocyte proteins as indicators of haemolysis, and proposed using them to flag compromised samples. In a clinical cohort where cases and controls were collected at different sites or at different times, these effects can generate differences that look like disease biology and are nothing of the kind. At the other extreme, the analysis of very small samples, from a few thousand cells down to single cells, turns sample loss into the dominant problem. Every surface a peptide touches adsorbs some of it, and at nanogram inputs the fraction lost to tubes, tips and columns can exceed the fraction measured. Low-input workflows therefore minimise transfers and volumes, performing lysis, digestion and sometimes labelling in the same nanolitre-scale well, and avoid clean-up steps where possible by using mass-spectrometry-compatible reagents. Single-cell proteomics, which has grown rapidly since the SCoPE-MS method was published by Bogdan Budnik, Nikolai Slavov and colleagues in 2018, relies on such miniaturised preparation. It is a separate field in its own right, but its lessons generalise: the less material there is, the more the preparation, rather than the instrument, sets the limit of detection. Designing the preparation, not just performing it Several design principles follow from this chapter, and they recur throughout the rest of the book. The first is that preparation defines the population being measured. A lysis buffer that fails to solubilise membranes produces a proteome depleted of membrane proteins, and no downstream method can recover them. When comparing studies, or when combining datasets, differences in preparation are often larger than differences in instrumentation. The second is that preparation is a source of batch effects. Samples prepared on different days, by different people, with different lots of trypsin or in different positions on a plate will differ systematically. The defence is randomisation and blocking: samples from each experimental group should be distributed across preparation batches so that batch and biology are not confounded, and the preparation order should be recorded so that batch can be modelled statistically later. A study in which all control samples were prepared in week one and all treated samples in week two cannot be rescued by any analysis. The third is that preparation choices interact with the acquisition strategy. Isobaric labelling requires that peptides be dissolved in an amine-free buffer, such as HEPES or triethylammonium bicarbonate, at a pH around eight to eight and a half, because TMT reagents react with primary amines; Tris buffer, which carries its own primary amine, will consume the reagent. Labelling efficiency must be checked, typically by a small test run searched with the tag set as a variable rather than fixed modification, because incompletely labelled peptides are invisible to the quantification. Label-free and DIA workflows avoid those constraints but depend even more heavily on run-to-run consistency of the digest, because every sample is measured separately and any difference in preparation propagates straight into the quantities. The fourth is that quality control starts here, not at the instrument. A standard digest, such as a commercial human cell-line digest, run at intervals alongside the study samples shows whether the chromatography and the instrument are stable; a pooled sample made from a small aliquot of every study sample, carried through preparation in each batch, shows whether the preparation is stable. The pooled reference plays an additional role in isobaric labelling designs, as a bridge between multiplexed sets, and Chapter 4 returns to it. Sample preparation is unglamorous, and its methods papers are less cited than those describing new instruments. But the depth, precision and validity of everything that follows are bounded by what happens in the first few hours of the workflow. The next chapter takes the peptide mixture that preparation produces and asks how to spread it out in time so that the mass spectrometer can see as much of it as possible. Chapter 2: Separation as a Resource A tryptic digest of a human cell line contains hundreds of thousands of distinct peptide species, spread across a concentration range of five or six orders of magnitude. If the whole mixture were sprayed into a mass spectrometer at once, the instrument would see only a dense forest of overlapping peaks dominated by the most abundant peptides, and the rest would be suppressed in the electrospray or lost beneath the noise. Separation solves this by spreading the mixture out in time, so that at any given moment only a manageable subset of peptides enters the instrument. It is best thought of not as a preliminary step but as a resource with a budget, measured in peak capacity, in instrument hours and in sample consumed. How that budget is spent determines the depth of the experiment and, increasingly, its throughput. Online chromatography: the conveyor belt into the instrument Almost all bottom-up proteomics couples reversed-phase liquid chromatography directly to electrospray ionisation. Peptides are loaded onto a column packed with small silica particles bearing C18 alkyl chains, in a mostly aqueous acidic solvent where they stick to the hydrophobic surface. A gradient of increasing organic solvent, usually acetonitrile, then elutes them roughly in order of hydrophobicity. The column outlet leads to a fine emitter at high voltage, where the liquid is dispersed into charged droplets that evaporate to release gas-phase peptide ions. Two properties of this arrangement shape everything else. The first is that electrospray behaves, over the relevant range, as a concentration-sensitive process: the signal depends on the concentration of analyte in the droplets rather than on the total amount delivered per unit time. Reducing the flow rate and the column diameter concentrates each peptide into a smaller volume and produces smaller initial droplets that ionise more efficiently. This is the reason the field adopted nanoflow chromatography, with columns of around 75 micrometres internal diameter running at a few hundred nanolitres per minute, as its default for discovery work. The sensitivity gain over conventional analytical flow is large, and for limited samples it is decisive. The second property is that chromatographic resolving power is finite. The useful measure is peak capacity, roughly the number of peaks that can be separated side by side within the gradient. It increases with column length, with smaller particle sizes that require higher pressures, and with longer gradients, but with diminishing returns: doubling gradient time does not double the number of peptides identified, because the gain in separation is partly offset by broader peaks and diluted signals. A typical nanoflow setup running a two-hour gradient on a 25 to 50 centimetre column may have a peak capacity of several hundred. Against a mixture of hundreds of thousands of peptides, that means many peptides still co-elute, and the mass spectrometer must sort them out by mass, by ion mobility and by fragmentation. Nanoflow's sensitivity comes at the cost of robustness. Nanolitre flow rates are hard to keep stable, the tiny emitters clog and degrade, retention times drift between runs and columns, and a single failure can interrupt a large sample series. For studies of hundreds or thousands of samples, where the limiting factor is not sample amount but reproducibility across weeks of continuous operation, several groups have argued for moving to higher flow. Yangyang Bian, Bernhard Kuster and colleagues, writing in Nature Communications in 2020, showed that microflow chromatography at several microlitres per minute, loading correspondingly more peptide, could deliver robust, reproducible analysis of thousands of proteomes with an acceptable loss of sensitivity. Clinical plasma proteomics projects in particular have moved toward higher flow rates and short gradients because they prize reliability over maximum depth per run. Throughput has become a design variable in its own right. A decade ago a typical deep proteome run lasted two to four hours. Dedicated chromatography systems now offer standardised methods defined by samples per day rather than gradient length. The Evosep One, described by Nicolai Bache, Philipp Geyer and colleagues in 2018, preforms the gradient at low pressure and stores it in a loop so that the high-pressure part of the run is short, which reduces the dead time between injections to a few minutes and allows standardised methods from tens to hundreds of samples per day. When paired with the fastest current instruments, short gradients of fifteen to thirty minutes now yield proteome depths that required hours of instrument time a few years ago. The Copenhagen group of Jesper Olsen, reporting on the Orbitrap Astral instrument in Nature Biotechnology in 2024, quantified about ten thousand human protein groups in half-hour runs and about seven thousand in five-minute runs using narrow-window DIA. Such figures are not typical of every laboratory or sample type, but they show how quickly the balance between depth and throughput has moved. Chromatography also introduces a subtle problem for quantification across runs. The retention time of a peptide shifts slightly between injections, and more between columns or laboratories. Label-free and DIA workflows depend on recognising the same peptide in every run, so they must align retention times. The indexed retention time, or iRT, concept introduced by Claudia Escher, Lukas Reiter and colleagues in 2012 expresses retention on a normalised scale anchored by a set of synthetic reference peptides spiked into every sample, which allows retention times from spectral libraries to be transferred to new runs. Modern software often uses endogenous peptides for the same purpose, and deep learning models now predict retention times directly from sequence with useful accuracy. Separation also governs a quieter effect that is easy to overlook: competition for charge in the electrospray. Peptides that elute together compete for the limited charge available on the surface of each droplet, and the more readily ionised species win. A peptide that ionises well in one sample may be partly suppressed in another if a highly abundant co-eluting peptide happens to be present there, for instance because the treatment induced a stress protein. Better separation reduces this competition by reducing the number of co-eluting species, which is one reason why the same peptide can give a more reliable quantity in a well-separated run than in a crowded one. It is also a reason why absolute signal intensities are not comparable between different peptides: each peptide has its own ionisation efficiency, determined by its sequence, and two peptides from the same protein, present at exactly the same molar amount, can differ in signal by more than an order of magnitude. Quantitative proteomics therefore almost always compares the same peptide across samples, rather than different peptides within a sample, and the methods for estimating absolute protein amounts from label-free data, such as summing intensities across all peptides of a protein or using only the most intense peptides, rely on averaging over this variation rather than removing it. Offline fractionation: buying depth with time When a single chromatographic dimension is not enough, the mixture can be split into fractions before online analysis, each of which is then run separately. The logic is simple. If the mixture is divided into twenty fractions that contain largely different peptides, each online run faces a far less complex mixture, and the instrument can sample deeper into each. The cost is equally simple: twenty times the instrument time per sample, and more handling. For fractionation to help, the offline separation must be orthogonal to the online one, meaning that it separates peptides by a property different from, or at least not strongly correlated with, hydrophobicity at low pH. The first widely used two-dimensional method was strong cation exchange, which separates peptides by charge. John Yates's group combined it with reversed-phase in a single capillary in the multidimensional protein identification technology, MudPIT, published by Michael Washburn, Dirk Wolters and John Yates in Nature Biotechnology in 2001, which identified well over a thousand yeast proteins at a time when that was remarkable. The method that displaced strong cation exchange for most deep proteome work is high-pH reversed-phase fractionation. It seems at first as though it should not work, since both dimensions are reversed-phase. The trick is that the selectivity of reversed-phase chromatography depends on the charge state of peptides, which changes with pH. At pH ten, acidic residues are deprotonated and basic ones largely neutral, so the elution order differs substantially from that at pH two or three. The orthogonality is imperfect, but the high-pH dimension has much higher resolving power than ion exchange and uses volatile, salt-free buffers that need no desalting. The remaining correlation between the dimensions is handled by concatenation, an approach described by Yuexi Wang and colleagues in 2011: the offline separation is collected into many small fractions, say ninety-six, and these are then pooled in a pattern in which early, middle and late fractions are combined, for instance fractions one, thirteen, twenty-five and so on into the first pooled fraction. Each pooled fraction then contains peptides spanning the whole hydrophobicity range of the second dimension, which uses the online gradient efficiently rather than concentrating peptides in one part of it. High-pH fractionation is how the deepest proteomes are made. Dorte Bekker-Jensen, Jesper Olsen and colleagues, in a 2017 Cell Systems paper, combined high-pH reversed-phase fractionation into dozens of fractions with fast data-dependent acquisition and reported identification of well over ten thousand protein-coding genes from human cell lines, approaching the complete expressed proteome. Such projects are resource maps rather than routine experiments, but they establish what is present to be measured and, importantly, they generate spectral libraries that make DIA analyses of single-shot runs more effective. Fractionation interacts strongly with the choice of quantification strategy, and this is one of the central design points of the book. In an isobaric labelling experiment, all samples in a multiplexed set are pooled before fractionation. One fractionated series of, say, twenty-four runs therefore delivers deep quantification for sixteen or eighteen samples at once. The instrument cost of fractionation is shared across the plex, which is why deep TMT experiments routinely fractionate and why they achieve depths that single-shot label-free analysis struggles to match. In a label-free or DIA experiment, each sample is measured separately, so fractionating each sample into twenty-four fractions multiplies instrument time for the whole study by twenty-four. For large cohorts that is prohibitive, and DIA studies therefore usually run each sample as a single shot and put their fractionation effort into building a library from a pooled sample, or into gas-phase fractionation, discussed below. A further design consideration is fraction-to-fraction variability. Because each fractionated sample is split into many runs, the quantity of a peptide measured in one sample depends on how it distributed across fractions, and small differences in fractionation between samples can introduce noise. In isobaric experiments this problem vanishes within a plex, because all samples experience the same fractionation physically together. In label-free designs with per-sample fractionation it must be controlled by summing signals across fractions and aligning carefully, which is one more reason large label-free studies avoid it. Separating in the gas phase Chromatography separates peptides in solution. Two further separation dimensions can act after ionisation, inside or just in front of the mass spectrometer, and both have become important in the last decade. Ion mobility separates ions by how quickly they drift through a gas under an electric field, which depends on their size, shape and charge. Two implementations dominate proteomics. High-field asymmetric waveform ion mobility spectrometry, or FAIMS, filters ions between the source and the mass spectrometer according to how their mobility changes between high and low fields; stepping through a small number of compensation voltages during a run acts as a form of online fractionation that removes singly charged background ions and reduces co-isolation. Alexander Hebert, Joshua Coon and colleagues showed in 2018 that FAIMS improved identification depth in single-shot analyses, and it has since been shown to reduce interference in isobaric labelling experiments as well. Trapped ion mobility spectrometry, or TIMS, as implemented on Bruker's timsTOF instruments, accumulates ions in a device where a gas flow is balanced against an electric field, then releases them in order of their mobility over tens of milliseconds. Coupled to a fast time-of-flight analyser, this allows parallel accumulation and serial fragmentation, or PASEF, in which the quadrupole jumps between precursors as they elute from the mobility device so that many precursors can be fragmented in a single mobility scan. Florian Meier, Matthias Mann and colleagues described PASEF in 2015 and 2018, and its extension to DIA, diaPASEF, in 2020. Ion mobility adds a coordinate to every peptide signal, and that extra coordinate is especially valuable for untangling the multiplexed spectra of DIA. Gas-phase fractionation is simpler and older. Rather than splitting the sample chemically, the same sample is injected several times and the instrument is told to examine a different slice of the precursor mass range each time. In DIA workflows this is an efficient way to build a deep, project-specific library: Brian Searle, Michael MacCoss and colleagues described in 2018 how a series of injections of a pooled sample, each covering a narrow mass range with very narrow isolation windows, could generate a chromatogram library that is directly matched to the experimental runs because it was collected on the same instrument under the same chromatography. Budgeting separation Several practical principles follow. Separation effort should be spent where the question needs it. A study seeking to catalogue low-abundance transcription factors in a cell line needs depth, which calls for fractionation, long gradients or both. A study of three hundred patient plasma samples seeking robust associations with a clinical variable needs reproducibility and throughput, which calls for short, robust single-shot runs, perhaps with higher flow. A time course with a modest number of samples and a need for complete data across all of them may be well suited to a single multiplexed set with extensive fractionation. The same instrument can serve all three, but not with the same method. Instrument time is a finite, expensive resource, and it should be counted at the design stage. A cohort of five hundred samples run in single-shot DIA at sixty samples per day consumes a little over a week of continuous acquisition, plus quality controls and blanks. The same cohort run with twenty-four fractions per sample on a two-hour gradient would need years. A deep multiplexed design would need roughly thirty sets of sixteen samples, each with its own fractionated series. Comparing these budgets at the planning stage, before any samples are prepared, prevents the common failure in which a study design is fixed first and discovered later to be unaffordable. Run order matters. Chromatographic performance drifts over hundreds of injections as columns age and emitters degrade, and carryover of abundant peptides from one run to the next is real. Samples should be injected in randomised or balanced order so that drift does not align with experimental groups, blanks should be interleaved where carryover is a concern, and quality-control standards should be run at regular intervals so that drift can be detected and, if necessary, corrected. Finally, separation cannot be optimised in isolation from acquisition. A short gradient produces narrow chromatographic peaks, perhaps a few seconds wide, which demands a fast instrument to sample each peak several times. A long gradient spreads peptides out and relaxes that constraint but reduces throughput. The next chapter turns to the mass spectrometer itself, and to the question of how an instrument that can examine only a few precursors at a time decides which ones to look at. Chapter 3: Inside the Mass Spectrometer A mass spectrometer does two things that matter for proteomics. It measures the mass-to-charge ratio of ions with great accuracy, and it can isolate chosen ions, break them apart and measure the pieces. The first operation, recorded as an MS1 or survey spectrum, says which peptide ions are present at a given moment and how intense they are. The second, recorded as an MS2 or tandem mass spectrum, provides the fragment pattern from which a peptide's sequence can be inferred. Every acquisition strategy, whether data-dependent, isobaric or data-independent, is a policy for deciding when to do which of these operations and on which ions. To understand why TMT and DIA behave as they do, it helps first to understand the machinery and the older policy they both set out to improve. Analysers, speed and fragmentation The core of a mass spectrometer is its mass analyser, the device that sorts ions by mass-to-charge ratio. Modern proteomics instruments are hybrids that combine two or three analysers, each doing what it does best. Table 1 summarises the main types in use for bottom-up work. Table 1. Mass analysers commonly used in bottom-up proteomics and their characteristic roles. Analyser Operating principle Main strength Typical role Quadrupole Oscillating electric fields pass a narrow m/z range Fast, selective isolation Precursor selection before fragmentation Linear ion trap Ions stored and ejected by m/z Sensitive, fast, flexible Fragment scans, MS3 selection Orbitrap Ions orbit a spindle electrode; frequency gives m/z High resolution and mass accuracy Survey scans, reporter ion and fragment scans Time-of-flight Flight time over a fixed path gives m/z Very fast acquisition Fragment scans, especially with ion mobility Astral Lossless multi-reflection time-of-flight path Speed with high sensitivity and resolution Fast MS2 for narrow-window DIA and DDA Three performance characteristics matter most. Resolution is the ability to distinguish ions of nearly identical mass-to-charge ratio. It determines whether two co-eluting peptides with similar masses appear as separate peaks and, critically for isobaric labelling, whether reporter ions that differ by a few thousandths of a dalton can be told apart. Mass accuracy is how close the measured mass is to the true mass; modern Orbitrap and time-of-flight instruments routinely achieve errors of a few parts per million, which sharply limits the number of candidate peptides that could explain a given precursor. Speed, expressed as the number of MS2 spectra that can be acquired per second, sets how many peptides can be sampled during the few seconds each one takes to elute. These properties trade against one another. In an Orbitrap, higher resolution requires a longer transient, the period over which the image current of the orbiting ions is recorded, and therefore a slower scan. Sensitivity also depends on how long ions are accumulated before each scan: collecting ions for longer gives better spectra of low-abundance peptides but slows the cycle. Instrument generations since the Orbitrap was introduced commercially in 2005 have progressively relaxed these trade-offs, and the introduction of the Orbitrap Astral in 2023, which pairs an Orbitrap for survey scans with the fast, sensitive Astral analyser for fragment scans at rates reported to exceed two hundred spectra per second, changed the practical ceiling on how many peptides can be fragmented in a run. Bruker's timsTOF line, pairing trapped ion mobility with time-of-flight, achieved high fragmentation rates by a different route. The competition between these platforms continues, and any statement about which instrument is best has a short shelf life. What endures is the logic: an acquisition strategy is a way to spend a fixed budget of ions and time. Fragmentation turns a precursor mass, which is compatible with many possible sequences, into a pattern that points to one. The dominant method is collision-induced dissociation, in which peptide ions are accelerated into an inert gas such as nitrogen or argon; the collisions deposit energy that breaks the peptide backbone, predominantly at the amide bonds. In the higher-energy collisional dissociation implemented on Orbitrap instruments, known as HCD, fragmentation takes place in a dedicated collision cell and all fragments are measured, including the low-mass region where TMT reporter ions appear. Earlier ion trap collision-induced dissociation had a low-mass cut-off that made reporter ions hard to observe, which is one reason HCD became the standard for isobaric labelling. When a peptide breaks at an amide bond, the charge can stay with either piece. Fragments that contain the amino terminus are called b ions, those that contain the carboxyl terminus y ions. A complete ladder of b or y ions spells out the sequence, since successive fragments differ by the mass of one amino acid residue. Real spectra are incomplete and noisy: some bonds rarely break, especially next to proline, some fragments carry multiple charges, neutral losses of water or ammonia add peaks, and the intensities of fragments vary in ways that depend on sequence. Tryptic peptides, with their basic carboxyl terminus, tend to produce strong y-ion series, which is one reason they are so easy to identify. Alternative fragmentation methods exist for specific problems. Electron transfer dissociation, developed by Donald Hunt, Joshua Coon, John Syka and colleagues in 2004, transfers an electron to a multiply charged peptide and breaks a different bond along the backbone, producing c and z ions. It tends to preserve labile modifications such as phosphorylation and glycosylation that collisional methods strip off, and it works well for highly charged, long peptides. Hybrid methods combine electron transfer with supplemental collisional activation. For deep quantitative profiling of unmodified peptides, however, HCD is the default, and the rest of this book assumes it unless otherwise stated. Data-dependent acquisition and its discontents For most of the history of shotgun proteomics, the policy for deciding what to fragment was data-dependent acquisition. The instrument records a survey scan, detects the peptide-like features in it, and selects the most intense, typically the top ten to twenty or as many as it can fit into a fixed cycle time. For each, the quadrupole isolates a narrow window of mass-to-charge ratio around the precursor, usually one to two units wide, the ions in that window are fragmented, and the MS2 spectrum is recorded. The cycle then repeats. Dynamic exclusion prevents the same precursor from being picked again for a set period, typically tens of seconds, so that the instrument moves on to less abundant species rather than repeatedly fragmenting the same abundant ones. Filters reject singly charged ions, which are mostly contaminants rather than tryptic peptides. The strengths of the approach are real. Each MS2 spectrum is, in principle, derived from a single precursor whose mass is known, which makes it easy to interpret. Search engines designed in the 1990s were built for exactly this kind of data, and the database search methods described in Chapter 6 remain the backbone of identification. The weaknesses emerged as the field tried to scale. The first is undersampling. A single shotgun run contains far more peptide species than the instrument can fragment. Annette Michalski, Jürgen Cox and Matthias Mann estimated in a 2011 Journal of Proteome Research paper that more than one hundred thousand peptide features were detectable in the survey scans of a single run of a complex human digest, but that the majority were inaccessible to data-dependent fragmentation, either because the instrument did not have time to pick them or because they were too weak to yield identifiable spectra. The run-to-run consequence is stochastic sampling. Which peptides reach the top of the intensity list at the moment the instrument is looking depends on small variations in chromatography and ion accumulation. A moderately abundant peptide may be fragmented in one run and not in the next. Across a study of many samples this produces a data matrix riddled with missing values, not because the peptide is absent from the sample but because the instrument happened not to select it. The second weakness is co-isolation. The isolation window around the selected precursor is narrow, but at any moment in a complex run it usually contains other peptide ions as well, some of them near the noise level of the survey scan. Their fragments contaminate the MS2 spectrum. Such chimeric spectra reduce identification scores and, when the quantity is read from the fragment spectrum, as in isobaric labelling, contaminate the measurement. This problem is central to Chapter 4. A short calculation makes the constraints concrete. Suppose a peptide elutes from the column over a chromatographic peak about fifteen seconds wide at its base, which is typical of a one- to two-hour nanoflow gradient; on a short high-throughput gradient the peak may be only five seconds wide. At any moment during a busy part of the gradient, perhaps several hundred peptide features are visible in the survey scan. If the instrument can acquire twenty fragment spectra per second, and each cycle consists of one survey scan followed by as many fragment scans as fit into a second, then in the fifteen seconds that one peptide is eluting the instrument can fragment around three hundred precursors, most of them only once because of dynamic exclusion. That sounds generous until one recalls that the same fifteen seconds contain several hundred to a thousand co-eluting features, many of them too weak to yield a useful spectrum with a short ion accumulation time. Doubling the speed of the instrument helps, but if the injection time must be shortened to achieve that speed, weak precursors produce even poorer spectra. The same arithmetic applies differently to quantification. To integrate a chromatographic peak reliably from MS1 survey scans, the peak must be sampled several times; a common rule of thumb in targeted and DIA work is at least six to ten points across the peak. With a fifteen-second peak, that requires a survey scan every two seconds or faster, which is easy. With a five-second peak it requires a cycle time under a second, which constrains how much fragmentation can be fitted between surveys. The move toward short gradients for throughput therefore pushes instruments toward higher speed, and it is no accident that the fastest current analysers were developed at the same time as high-throughput chromatography. Ion accumulation introduces a second constraint. Orbitrap instruments collect ions in a trap before each scan, controlled by a target number of charges called the automatic gain control setting and a maximum injection time. The target exists because too many ions in the analyser distort the measurement through space-charge effects, degrading mass accuracy. The consequence is that each spectrum has a finite dynamic range: if a very abundant ion fills most of the available capacity, weaker ions in the same scan are represented by few charges and measured imprecisely. This is one reason why dynamic range within a single survey scan is several orders of magnitude narrower than the dynamic range of the proteome, and why separation, which reduces the number of ions competing for each scan, matters so much. Label-free quantification in data-dependent mode Even without labels, data-dependent runs can be used to quantify. The crudest method, spectral counting, simply counts how many MS2 spectra were matched to each protein, on the reasoning that abundant proteins yield more peptides and more repeated selections. It is easy and surprisingly informative for large differences, but it saturates for abundant proteins, is noisy for scarce ones and is distorted by dynamic exclusion. The more precise approach uses the MS1 signal: once a peptide is identified from its MS2 spectrum, the area under its chromatographic peak in the survey scans is integrated, and that area serves as its quantity. Peptide quantities are then combined into protein quantities. To address missing values, software can transfer identifications between runs. The widely used MaxQuant package implemented this as match between runs: if a peptide was identified by MS2 in one run, the software looks in other runs for an MS1 feature with the same accurate mass and a retention time that, after alignment, is close enough, and assigns the identity even though no MS2 spectrum was taken. The MaxLFQ algorithm described by Jürgen Cox and colleagues in 2014 then computes protein intensities from pairwise peptide ratios in a way designed to be robust to missing values and to fractionation. Match between runs substantially reduces missing values, but it introduces a new kind of error. A transferred identification rests on mass and retention time alone, and in a crowded run there may be several features that fit. Controlling the error rate of these transfers has been a recurring concern, and several groups have proposed methods to estimate it, for instance by transferring decoy identifications alongside real ones and counting how often they are matched. What the discontents set in motion The two strategies that are the core of this book can be understood as different answers to the problems of data-dependent acquisition. Isobaric labelling accepts the data-dependent paradigm, including its stochastic sampling, but changes what a single selection achieves. By pooling many samples before analysis, it ensures that whenever a peptide is picked, it is quantified in all of them simultaneously. The missing-value problem is solved within a multiplexed set, though it reappears between sets. Co-isolation, however, becomes more damaging, because the quantity itself now comes from the fragment spectrum. Data-independent acquisition abandons the idea of choosing precursors. By fragmenting everything in systematic windows, it guarantees that every detectable peptide is sampled in every run, removing stochastic sampling at the acquisition stage. The price is that co-isolation is no longer an occasional contamination but the normal state of every spectrum, and the burden of sorting fragments to their precursors shifts entirely to software. Both answers are clever, both have limitations, and the next two chapters examine each in turn. The distinction between them is sometimes presented as labelled versus label-free, but that obscures more than it reveals. The deeper distinction is between reducing the number of runs and making every run complete. Hybrid methods that combine labelling with data-independent acquisition, discussed in Chapter 5, show that the two ideas are not mutually exclusive. One final point about instruments deserves emphasis, because it affects how published comparisons should be read. Each acquisition strategy was developed on, and optimised for, particular instrument architectures, and its relative performance changes with each hardware generation. Isobaric labelling with synchronous precursor selection and MS3 was developed on Orbitrap tribrid instruments that have a linear ion trap; diaPASEF was developed on timsTOF instruments; narrow-window DIA was enabled by the Astral analyser. A comparison between methods performed on one instrument generation is informative about the principles but not always about the present. Readers evaluating the literature, or planning a study, should ask which instrument was used, in what year, and whether the conclusion depends on a speed or sensitivity limit that has since moved. Hashtags: #ProteomicsByDesign #QuantitativeProteomics #BottomUpProteomics #TandemMassTags #TMTProteomics #IsobaricLabeling #DataIndependentAcquisition #DIAProteomics #DataDependentAcquisition #MassSpectrometry #ShotgunProteomics #RatioCompression #SPSMS3 #SpectralLibraries #LibraryFreeDIA #IonMobility #DiaPASEF #ProteinQuantification #PeptideIdentification #TargetDecoyStrategy #FalseDiscoveryRate #QuantitativeBioinformatics #DifferentialAbundance #ProteomicsExperimentalDesign #FutureOfQuantitativeProteomics

Download the Book (PDF): Introduction A mass spectrometer does not read proteins. It weighs fragments of them. When a peptide is broken apart inside the instrument, the resulting spectrum is a list of masses and intensities, and nothing in that list announces which gene the peptide came from. To turn a spectrum into a biological statement, a computer compares it with the spectra that would be expected from a set of candidate sequences and reports the best match. The candidates come from a file, a protein sequence database, which someone had to decide to build in a particular way. That file is the quiet centre of this book. For most of the history of shotgun proteomics, it was taken for granted. A laboratory downloaded the reference proteome for its organism, searched its spectra against it, and reported the proteins it found. This works well for a great deal of biology, and it will continue to. But it builds in an assumption that is easy to forget: that every protein worth finding is already written down. A peptide that is not in the database cannot be identified, however clean its spectrum. It will either go unmatched or, worse, be assigned to the closest sequence that is in the file, and reported as something it is not. Proteogenomics is the discipline that refuses this assumption. It uses genomic and transcriptomic data, usually from the same sample the proteins came from, to write a better database: one that contains the patient's own single-nucleotide variants, the splice junctions actually used in that tissue, the transcripts assembled from its RNA, the open reading frames that ribosomes were observed to translate. Mass spectrometry then serves as the arbiter. If a peptide carrying a predicted amino-acid change is detected, the variant is not merely present in DNA; it is expressed as protein. If a peptide spans an exon junction never seen in the reference annotation, a new isoform is supported at the protein level. If a peptide maps to a region annotated as non-coding, the annotation is challenged. The term covers two traffic directions. In one, protein data correct and extend genome annotation: peptides are used to confirm gene models, find missed genes and fix start sites. This was the original meaning, and it remains central to annotating newly sequenced organisms. In the other, genomic data make proteomics more complete and more personal: sample-specific sequences are added so that the proteins peculiar to one tumour, one cell line or one person can be seen at all. The cancer-focused work of the past decade, including the large programmes that profiled hundreds of tumours with both sequencing and deep proteomics, belongs mainly to this second direction. In practice the two meet in the same computational pipeline. The argument of this book The controlling idea here is simple to state and easy to underestimate. In proteogenomics, the sequence database is the hypothesis, and every discovery is only as trustworthy as the design of that database and the validation that follows. Adding sequences buys the possibility of discovery. It also costs statistical power, invites false matches and opens the door to artefacts that masquerade as biology. The whole craft lies in managing that trade: expanding the search space far enough to see what the reference misses, and no further than the evidence can support, then testing every surprising match as if it were wrong until shown otherwise. This framing matters because the field has repeatedly been burned by forgetting it. Early six-frame translation searches reported novel peptides that later proved to be modified or mis-assigned canonical ones. Claims about how much of the immunopeptidome arises from exotic processing mechanisms were revised sharply once the search strategies behind them were scrutinised. Headline counts of "novel proteins" have tended to shrink when stricter evidence standards were applied. None of this means proteogenomics is unreliable. It means the reliability is manufactured, step by step, by choices that a careful analyst can make well or badly. What the book covers The chapters follow the logic of a real analysis. The first explains why reference proteomes are incomplete, and why that incompleteness is structured rather than random: it is concentrated in short proteins, non-canonical reading frames, individual variation and tissue-specific isoforms. The second explains how mass spectrometry actually identifies peptides, with particular attention to the target-decoy method and the false discovery rate, because everything later depends on understanding what a 1 percent threshold does and does not promise. The third chapter turns to construction: how variant calls, RNA sequencing assemblies, splice-junction graphs and ribosome profiling are converted into protein sequences, and what each source contributes and costs. The fourth faces the statistical consequence of larger databases, explaining why a novel peptide passing a global threshold can still be much more likely to be wrong than a canonical one, and what the accepted remedies are. Four chapters then follow the applications. One concerns alternative splicing and the real, somewhat sobering evidence about how many alternative isoforms are detectable as protein. One concerns non-canonical proteins: the short open reading frames, upstream reading frames and translated "non-coding" RNAs that ribosome profiling has revealed, and the difficulty of confirming them by mass spectrometry. One reviews what the large tumour cohort studies, which measured genome, transcriptome, proteome and phosphoproteome in the same specimens, actually taught about how genomic change reaches the protein level. One concerns personalised tumour antigens, where the stakes are clinical, the peptides are displayed on human leukocyte antigen molecules rather than produced by trypsin, and the question is not only whether a peptide exists but whether a T cell can see it. The final chapter gathers the validation protocols that apply across all three: peptide-centric re-searching against every competing explanation, spectral comparison with synthetic standards and predicted spectra, retention-time consistency, and a tiered way of stating how confident a claim deserves to be. Who this is for The reader imagined here is scientifically literate but not necessarily a specialist in both halves of the field. Many readers arrive from one side: a genomicist who has variant calls and wants to know whether they are expressed; a proteomicist who has unexplained spectra and wants to know what they might be; a cancer immunologist who wants to know which neoantigen predictions to trust; a bioinformatician asked to build the pipeline. The book tries to give each enough of the other side to reason well, without pretending to be a software manual. Tools are named where they illustrate a principle, and the principles are what should outlast the tools. No knowledge of statistics beyond the idea of a threshold is assumed, but the book does not avoid the statistics, because the statistics are where most proteogenomic claims succeed or fail. Nor does it avoid the chemistry of fragmentation and the arithmetic of residue masses, because a surprising number of apparent discoveries dissolve once someone checks whether two amino acids, or an amino acid and a modification, weigh the same. A note on evidence Proteogenomics sits at the join between two data types that are each noisy in their own ways. Sequencing errors, alignment artefacts and mis-called variants propagate into the database; spectral noise, co-isolation and unexpected modifications propagate into the matches. When the two are combined, errors do not cancel. They compound, because a false sequence in the database gives a noisy spectrum something to be falsely matched to. The recurring advice in these pages is therefore conservative: prefer sample-specific evidence to generic expansion, report novel findings under separate and stricter thresholds, and treat every novel peptide as a claim requiring a second, independent line of support. This is not caution for its own sake. The payoff of the discipline, when it is practised well, is considerable. It has shown which cancer mutations reach the protein level and which do not, revealed signalling consequences of somatic alterations that genomics alone could not predict, supported the translation of reading frames that annotation had missed, and identified tumour-specific peptides actually displayed to the immune system. Those results are credible because the people who produced them built their databases carefully and tested their matches hard. The chapters that follow set out how. Chapter 1: The Incomplete Reference Every protein sequence database is a model of what a cell can make. The reference proteomes distributed by UniProt, Ensembl and RefSeq are extraordinarily good models, built over decades from cDNA cloning, comparative genomics, manual curation and, increasingly, protein-level evidence. They are also models of a particular kind: a consensus drawn from many individuals, weighted toward long, conserved, well-studied proteins, and organised around rules for what counts as a gene. Understanding where those rules leave gaps is the first step in deciding what a customised database should add. How genes get annotated Genome annotation is an inference from evidence. Annotators align transcripts and known proteins to the genome, look for open reading frames with plausible start and stop codons, use conservation across species to judge whether a reading frame is under selection for its protein product, and then decide. The GENCODE project, which provides the reference gene set used by Ensembl for human and mouse, combines automated pipelines with manual curation by the HAVANA group, and its releases describe tens of thousands of protein-coding and non-coding genes and a much larger number of transcripts. Two design choices in this process shape proteogenomics directly. First, annotation historically required a minimum open reading frame length, commonly around 100 codons, to call a protein-coding gene without additional evidence. The reason was statistical: random sequence produces short open reading frames by chance all the time, so length was a cheap filter against false genes. The cost was that genuine short proteins were systematically excluded. Second, annotation relied heavily on evolutionary conservation. A reading frame whose protein-coding character was conserved across mammals was easy to accept; a young, lineage-specific or poorly conserved one was not. Both filters are sensible, and both create blind spots in exactly the places proteogenomics now looks. Protein-level evidence has itself reshaped the gene count. Ezkurdia and colleagues, combining large proteomics datasets with conservation and other evidence, argued in 2014 that there may be as few as 19,000 human protein-coding genes, fewer than annotations then listed, because many annotated coding genes showed no peptide evidence and features suggesting they were not truly coding. The point is not the precise number but the direction: annotation is revised in both directions, removing genes that fail to earn their status as well as adding new ones, and mass spectrometry is one of the tools that does the revising. What mass spectrometry has already seen In 2014 two groups published draft maps of the human proteome in the same issue of Nature. Kim and colleagues profiled a panel of adult and fetal tissues and primary cells; Wilhelm and colleagues assembled a large compendium of their own and public data into the ProteomicsDB resource. Together they showed that deep, tissue-diverse shotgun proteomics could detect peptides from the large majority of annotated protein-coding genes. They also reported peptides that did not match the reference: some from putative non-coding regions, some from alternative reading frames, some from unannotated exons. Several of those claims were later questioned, which is itself instructive, but the broad lesson held. A deep proteome contains signal that the reference database cannot explain. The Human Proteome Project, coordinated by the Human Proteome Organization, has tracked this systematically through the neXtProt knowledgebase, grading each human protein by its level of existence evidence. By 2020, Adhikari and colleagues reported that credible mass spectrometry or other protein-level evidence had been obtained for more than 90 percent of the predicted human proteins. The remaining "missing proteins" are not randomly distributed: they include olfactory receptors and other proteins expressed in few cells, membrane proteins with few tryptic sites, and proteins expressed only in specific developmental stages or tissues. The project's guidelines for claiming detection, discussed in the final chapter, are some of the most useful statements anywhere of what evidence a novel protein claim requires. Four kinds of gap For the purposes of database design it helps to sort the gaps in the reference into four kinds, because each is filled from a different data source and each carries a different risk. Individual variation. The reference proteome is essentially one sequence per protein, reflecting a reference genome assembled from a small number of donors. Any real individual differs from it at millions of positions, a few tens of thousands of which fall in coding sequence and change an amino acid. Population resources such as gnomAD catalogue these variants across many thousands of people. A peptide covering a missense variant will not match the reference sequence exactly, so in a standard search it goes unidentified or is matched incorrectly. For most applications the loss is small, since other peptides from the same protein are still found. For some it is the whole point: the protein product of a germline disease allele, or a somatic mutation in a tumour, is precisely the peptide a standard search cannot see. Somatic alteration. Cancer genomes add their own changes: point mutations, small insertions and deletions that shift the reading frame, gene fusions that join two coding sequences, and large-scale rearrangements. Frameshift insertions and deletions produce stretches of entirely novel amino-acid sequence downstream of the event, often until a new stop codon is reached. Fusions produce junction peptides that exist in no normal cell. These are the raw material of tumour-specific antigens, and none of them appear in any reference. Isoform diversity. Most human multi-exon genes produce more than one transcript through alternative splicing, alternative promoters and alternative polyadenylation. Reference databases include many annotated isoforms, but not every junction used in a given tissue, and certainly not the aberrant splicing that is common in tumours, where mutations in splicing factors such as SF3B1 generate recurrent novel junctions. A peptide that spans a novel exon-exon junction is invisible unless the junction is in the database. Non-canonical translation. Ribosomes translate more than annotated coding sequences. They initiate at upstream open reading frames in 5′ untranslated regions, at near-cognate start codons such as CUG, in reading frames overlapping known coding sequences, and within RNAs annotated as long non-coding. Many of the resulting products are short, some are unstable, and some have demonstrable functions. They are the least represented class in reference databases, partly because of the length and conservation filters described above. Why transcripts are not proteins A natural response to these gaps is to ask why proteomics is needed at all. If RNA sequencing shows the variant allele, the novel junction or the translated reading frame, why not accept it? The answer is that each step between DNA and functional protein is regulated and lossy. A somatic variant can be present in DNA but lie in an allele that is not expressed. A transcript can be abundant but subject to nonsense-mediated decay, or retained in the nucleus, or translated poorly. A reading frame can be engaged by ribosomes, as ribosome profiling shows, yet yield a product degraded almost as soon as it is made. Large studies that measured both mRNA and protein in the same tumours have repeatedly found that the correlation between them, gene by gene across samples, is positive but modest and highly variable. The first large proteogenomic study of colorectal cancer from the Clinical Proteomic Tumor Analysis Consortium, published by Zhang and colleagues in 2014, found that mRNA levels were limited predictors of protein abundance, and that many copy-number alterations visible in the genome had surprisingly weak effects at the protein level, with notable exceptions in a small number of amplified regions. Proteins are also the molecules drugs bind and immune receptors recognise. A T cell does not see a mutation; it sees a short peptide displayed by a human leukocyte antigen molecule on the cell surface. A kinase inhibitor does not act on a gene; it acts on a folded protein, whose activity may be set by phosphorylation that no sequencing assay measures. For questions like these, protein-level evidence is not a confirmation of the genomic result but a different and often more decisive observation. Why proteins cannot simply replace sequencing The complementary question is why proteomics needs genomics. The answer lies in coverage and sensitivity. A shotgun experiment samples peptides stochastically and favours abundant proteins; the dynamic range of protein concentrations in a cell or tissue spans many orders of magnitude, and in plasma considerably more. For any given protein only a subset of its peptides is observable, because some are too short or too long after digestion, some ionise poorly, and some carry modifications. The probability that the one peptide covering a specific variant, junction or novel reading frame is among those detected is often low. Mass spectrometry also cannot easily distinguish sequences that are identical in mass. Leucine and isoleucine have the same elemental composition; glutamine and lysine differ by only about 0.036 daltons; the dipeptide glycine-glycine has the same composition as asparagine. Without a candidate sequence to test, many spectra are consistent with several sequences. De novo sequencing, which reads peptide sequences directly from spectra without a database, has improved markedly with deep learning, but it still struggles with incomplete fragmentation and these ambiguities, and it is most powerful when combined with a database rather than replacing one. So each technology supplies what the other lacks. Sequencing proposes what could exist, with high sensitivity and near-complete coverage of the transcribed genome. Mass spectrometry tests which of those possibilities exist as protein, with lower sensitivity but direct physical observation of the molecule. The union is more informative than either, but only if the proposal step does not overwhelm the testing step, a tension that runs through every chapter that follows. Beyond the human genome It is worth remembering that proteogenomics began not with personalised medicine but with genome annotation of organisms for which no mature reference existed. When a bacterial, fungal, plant or parasite genome is first sequenced, its gene models are predicted computationally, and prediction errors are common: missed small genes, wrong start codons, genes predicted on the wrong strand, frameshifts introduced by sequencing errors in homopolymer runs. Searching the organism's own proteome against a translation of its whole genome allows peptides to vote on the gene models. A peptide found upstream of a predicted start codon indicates that translation begins earlier; a peptide mapping to an unannotated intergenic region suggests a missed gene; a set of peptides that straddle a predicted frameshift suggests the sequence, not the organism, is at fault. In compact prokaryotic genomes this strategy is powerful because the search space is small. A few million bases translated in six frames yields a database of manageable size, and most of the genome codes for protein anyway, so the expansion from the predicted proteome to the full translation is modest. The approach has been used to refine annotations for many bacteria and archaea, and it pairs naturally with methods that enrich protein amino termini, which pinpoint where translation actually starts. The same strategy applied to the human genome, which is several hundred times larger and overwhelmingly non-coding, behaves very differently, a contrast that will matter a great deal in the chapters on database construction and statistics. What counts as novel The word "novel" carries a lot of weight in proteogenomic papers, and it helps to be exact about it before going further. A peptide can be novel relative to the particular database used for a standard search, relative to all major public references, or relative to the scientific literature. These are different claims. A peptide absent from a curated canonical database may be present in an isoform database, in a variant database or in an older annotation release; a peptide absent from all current references may already have been reported in a ribosome profiling study or a previous proteogenomic analysis. A second distinction concerns what the novel peptide implies. Some novel peptides imply only a small change to a known protein: a single amino-acid substitution, a different exon junction, an extended amino terminus. Others imply an entirely new protein product: a reading frame in an untranslated region, in a long non-coding RNA or in an alternative frame of a known gene. Still others imply a molecular event that exists only in the sample: a somatic frameshift or a fusion junction. Each category carries a different prior probability of being real, and the book will argue repeatedly that this prior should shape how much evidence is demanded. A variant peptide predicted from a high-confidence DNA call and supported by RNA reads is a modest claim. A peptide implying a previously unknown protein from a non-coding region is an extraordinary one. Finally, novelty at the peptide level does not automatically transfer to the protein level. A single peptide shows that a particular sequence was present in the sample; it does not show which full-length protein it came from, whether that protein is stable, or whether it does anything. Much of the careful language in the proteogenomic literature, such as speaking of "peptide evidence for translation" rather than "a new protein", reflects this gap. It is a good habit to adopt from the start. The personal proteome One more gap deserves emphasis because it motivates the most ambitious applications. Every individual, and every tumour, has a proteome that differs from the reference in ways that matter to that individual. A pharmacogenomic variant may change an enzyme's active site; a germline variant may create or destroy a cleavage site that changes which peptides are produced; a tumour's somatic mutations may create epitopes that its immune system can target. A sample-specific database, built from that sample's own genome and transcriptome, represents this personal proteome directly. The idea is appealing and the benefits are real, but it also shifts the burden of proof. A reference database has been checked by many people over many years. A personal database was built last week by one pipeline from one sequencing run. Its errors are private too. Every variant call, every assembled transcript and every predicted junction is a candidate, not a fact, and the proteomic search is only one of several tests it must pass. Holding both halves of that picture in view, the promise of seeing the personal proteome and the fragility of the database that makes it visible, is the habit of mind proteogenomics requires. Chapter 2: How a Spectrum Becomes a Peptide Before deciding what to put in a database, it is necessary to understand exactly how the database is used. The details of peptide identification are often treated as a black box by people who use its output, and that is where most proteogenomic mistakes begin. This chapter walks through the path from protein sample to reported peptide, concentrating on the points where the database exerts its influence. From protein to spectrum In a typical bottom-up experiment, proteins extracted from cells or tissue are denatured, their cysteines reduced and alkylated, and the mixture digested with trypsin. Trypsin cuts after lysine and arginine, except, as a rule, when the next residue is proline. The result is a very complex mixture of peptides, most between about seven and thirty residues long, which is separated by reversed-phase liquid chromatography and sprayed into the mass spectrometer by electrospray ionisation. The instrument performs two kinds of measurement in alternation. A survey scan, called MS1, records the mass-to-charge ratios of the intact peptide ions eluting at that moment. The instrument then selects precursor ions, isolates them within a narrow window, fragments them, usually by collisions with gas in a process called higher-energy collisional dissociation, and records the masses of the fragments in an MS2 scan. Peptide bonds tend to break along the backbone, producing two series of ions: b ions, which contain the amino terminus, and y ions, which contain the carboxy terminus. The mass differences between successive ions in a series correspond to residue masses, so a complete series spells out the sequence. Real spectra are rarely complete. Some bonds break reluctantly, especially next to proline or in certain residue pairs. Neutral losses of water or ammonia add peaks. Other peptides co-isolated in the same window contribute fragments of their own. Modifications, whether biological such as phosphorylation or introduced during sample preparation such as oxidation of methionine, shift masses. A single experiment on a modern instrument records tens of thousands to hundreds of thousands of MS2 spectra, and a sizeable fraction of them are never confidently identified. In data-dependent acquisition, the mode described above, the instrument chooses which precursors to fragment on the fly, usually the most intense. In data-independent acquisition, it instead fragments everything within successive wide windows across the mass range, producing composite spectra that must be deconvolved computationally. Data-independent methods give more consistent quantification across samples and have become common in large cohort studies, but their identification strategies depend even more heavily on prior knowledge of what peptides to look for, typically in the form of spectral libraries or predicted spectra. Most proteogenomic discovery has so far been done with data-dependent acquisition, for reasons that will become clear. Database searching The dominant method for identifying peptides from MS2 spectra is sequence database searching. The search engine digests every protein in the database in silico according to the enzyme rules, computes the mass of every resulting peptide, and for each observed spectrum retrieves the candidate peptides whose masses lie within a tolerance of the measured precursor mass. For each candidate it predicts the fragment ions expected, compares them with the observed peaks and computes a score. The best-scoring candidate becomes the peptide-spectrum match, or PSM, for that spectrum. Several parameters control how many candidates each spectrum faces. The precursor mass tolerance on modern high-resolution instruments is typically a few parts per million, which is narrow; the number of missed cleavages allowed, usually two, multiplies the number of peptides per protein; and every variable modification allowed, such as methionine oxidation or amino-terminal acetylation, multiplies candidates further. Semi-specific or non-specific enzyme settings, needed when peptides do not arise from clean tryptic cleavage, expand the space by an order of magnitude or more. This matters because the score of the best match is only meaningful relative to the scores of the competitors it beat. The more candidates a spectrum is compared with, the higher the best random score it will encounter. Search engines differ in how they score. Early tools such as SEQUEST used cross-correlation between observed and predicted spectra; Mascot uses a probability-based score; X!Tandem, Comet, MS-GF+ and Andromeda in MaxQuant each have their own scoring functions and statistical models. MSFragger, introduced by Kong and colleagues in 2017, uses a fragment-ion index to make searches dramatically faster, which made practical the "open" searches that allow any mass shift on the precursor and thereby reveal unexpected modifications. That capability turns out to be important in proteogenomics, because many apparent novel peptides are really known peptides carrying a modification that was not in the search parameters. Other ways to identify a spectrum Database searching is not the only strategy. The main alternatives differ in what prior knowledge they need, and therefore in how they interact with a customised database, as Table 1 summarises. Table 1. Principal strategies for identifying peptides from tandem mass spectra. Strategy Prior knowledge required Strength Main limitation for proteogenomics Sequence database search Protein sequence database Sensitive, well-understood statistics Cannot find sequences absent from the database Spectral library search Library of previously identified spectra Fast, highly specific Library rarely contains novel peptides Open or mass-tolerant search Database plus wide precursor tolerance Reveals unexpected modifications Larger search space, harder error control De novo sequencing None beyond fragmentation rules Can propose sequences absent from any database Ambiguities and incomplete spectra limit accuracy Peptide-centric search A specific candidate peptide Direct test of one hypothesis against all alternatives Answers only the question asked Spectral library searching compares an observed spectrum with a library of previously identified experimental spectra and is fast and specific, but by construction a library contains only what has already been seen. Predicted spectral libraries partly remove that constraint. Prosit, described by Gessulat and colleagues in 2019, is a deep learning model trained on very large sets of synthetic peptide spectra that predicts fragment intensities and chromatographic retention times for arbitrary sequences, and similar models have followed. With such predictions, a novel peptide from a proteogenomic database can be given an expected spectrum and an expected elution time, both of which become powerful validation tools. De novo sequencing reads the sequence directly from the mass ladder. It needs no database and so is in principle ideal for finding the unexpected, and deep learning approaches have raised its accuracy considerably. In practice, full-length correct de novo sequences are obtained for a minority of spectra, and the method is most useful in proteogenomics as a complement: for proposing candidates in samples without matched sequencing data, or for checking whether a spectrum matched to a novel database entry independently supports that sequence. Peptide-centric searching inverts the usual question. Instead of asking which peptide best explains each spectrum, it asks, for one candidate peptide, whether any spectrum supports it better than every alternative explanation. Chapter 9 discusses PepQuery, the best-known implementation, as a validation step. The target-decoy approach A search engine will always report a best match, whether or not the true peptide is in the database. The central problem of peptide identification is therefore distinguishing correct from incorrect PSMs. A score threshold alone does not solve it, because the distribution of scores for incorrect matches depends on the data, the instrument, the database and the parameters. The solution that became standard is the target-decoy strategy, formalised by Elias and Gygi in 2007. Alongside the real protein sequences, the targets, the search includes an equal number of decoy sequences that cannot be correct: typically the target sequences reversed, or shuffled in a way that preserves amino-acid composition and, often, the enzymatic cleavage sites. Every spectrum is searched against the combined database. Because a spectrum that does not truly belong to any target peptide is, by assumption, equally likely to be matched by chance to a target or a decoy, the number of decoy matches above any score threshold estimates the number of incorrect target matches above that threshold. The false discovery rate is then estimated as the ratio of decoys to targets passing the threshold, and the threshold is chosen so that this ratio is at most, say, 1 percent. The method is simple, general and empirical, which is why it prevailed. But it rests on assumptions that proteogenomic databases can quietly violate. Decoys must be generated from the same database as the targets, with the same size and composition, so that random matches fall on them at the same rate. Every spectrum must be given the same chance to match a decoy as a target. And the false discovery rate it estimates is a property of the whole set of matches passing the threshold, not of any particular match. A 1 percent rate across a list of fifty thousand PSMs says that roughly five hundred are wrong; it says nothing about which ones. If the wrong ones are concentrated in a small subgroup, such as the novel peptides that are the reason for the experiment, the error rate within that subgroup can be very much higher than 1 percent. Chapter 4 is devoted to that problem. Rescoring and machine learning A single search-engine score uses only part of the information in a match. Other features also distinguish correct from incorrect PSMs: the precursor mass error, the number of matched fragment ions, the fraction of total intensity explained, the charge state, the number of missed cleavages, and the difference in score between the best and second-best candidate. Percolator, introduced by Käll and colleagues in 2007, uses a semi-supervised support vector machine that learns, iteratively from the target and decoy matches of each dataset, how to combine such features into a better discriminant score. It typically increases the number of identifications at a fixed false discovery rate. More recent rescoring adds features derived from predicted spectra and retention times: how well the observed fragment intensities agree with those predicted by a model such as Prosit for the candidate sequence, and how close the observed elution time lies to the predicted one. These features are especially valuable for peptides with unusual sequences, including non-tryptic peptides eluted from human leukocyte antigen molecules, where classic scores perform poorly. They carry a caution, however. Rescoring learns from the data what a correct match looks like, and if the training set is dominated by canonical peptides, novel peptides that behave differently may be treated unfairly in either direction. The safest practice is to check that rescoring behaves sensibly on the novel subset specifically, not only on the whole. From peptides to proteins The final step, protein inference, assembles peptide identifications into protein identifications. It is harder than it looks, because many peptides are shared between proteins: between isoforms of one gene, between members of a gene family, and, in proteogenomic databases, between a reference protein and its variant or alternative version. A peptide that is shared cannot by itself indicate which of its possible parent proteins was present. Protein inference algorithms apply parsimony, reporting the minimal set of proteins that explains the observed peptides, or probabilistic models that assign each protein a probability given the peptides. For proteogenomics, the lesson is that evidence for a novel product nearly always rests on peptides unique to that product: the variant-containing peptide, the novel junction peptide, the peptide from the unannotated reading frame. Shared peptides say that something from the locus was present, not that the novel form was. Because unique peptides are few, many novel claims rest on one or two PSMs, which is exactly why they require the extra scrutiny the rest of this book describes. What the database controls It is now possible to state precisely what the database does. It determines which sequences can be identified at all. It determines how many candidates each spectrum competes against, and so the distribution of chance scores and the threshold needed to control errors. It determines the decoys, and so the error estimate. It determines which peptides are unique and which are shared, and so what protein-level conclusions can be drawn. A proteogenomic analysis is, in effect, an experiment designed by building a database, and the rest of the pipeline reports the outcome. The next chapter examines how such databases are made. Chapter 3: Building the Customised Database A customised protein sequence database is assembled from pieces, each derived from a different kind of evidence and each translating nucleotide information into amino-acid sequences in its own way. The choices made here settle most of what the downstream search can and cannot find. This chapter goes through the main sources in rough order of how directly they reflect the sample: from generic translations of the genome, through population and sample-specific variants, to transcripts assembled from the sample's RNA and reading frames observed on its ribosomes. The spectrum of databases Proteogenomic databases differ in two properties that pull against each other: breadth, meaning how much of the possible proteome they include, and specificity, meaning how much of their content is actually likely to be present in the sample. As Table 2 sets out, the generic approaches maximise breadth at great cost in size, while sample-specific approaches keep the database small but can include only what the sequencing data detected. Table 2. Main sources of sequences for proteogenomic databases. Source Input data What it adds Relative size Main risk Six-frame genome translation Genome sequence Every possible reading frame Very large Severe loss of sensitivity, many false matches Three-frame transcript translation Annotated or assembled transcripts Alternative frames and untranslated regions Large Most frames are not translated Variant database DNA or RNA variant calls, or public catalogues Single amino-acid variants, small indels Small to moderate Mis-called variants, variants in unexpressed alleles Splice-junction database RNA-seq junction reads Novel exon-exon junction peptides Small to moderate Alignment artefacts, junctions from immature transcripts Transcript assembly RNA-seq reads Novel isoforms, fusions, unannotated genes Moderate Assembly errors, arbitrary reading-frame choice Ribo-seq ORFs Ribosome profiling Translated ORFs including short and upstream ORFs Small to moderate Ribosome occupancy without stable product Genome translation The oldest strategy is to translate the entire genome in all six reading frames, three on each strand, splitting at stop codons, and to search against every resulting stretch of amino acids above some minimum length. It requires no annotation and makes no assumptions, which is why it served well for early annotation of small genomes. For a human-sized genome it is, however, a poor default. The six-frame translation of the human genome is vastly larger than the reference proteome, and nearly all of it is never translated. Every spectrum faces orders of magnitude more candidates, the score needed to reach a given error rate rises sharply, and the identification rate falls. Worse, the novel matches that do pass are drawn from a region of search space where the prior probability of a real peptide is tiny, so a larger fraction of them are wrong than the global threshold implies. Six-frame searches of human data can still be useful for specific questions, and in a restricted form, for instance translating only regions with evidence of transcription, but they should not be the starting point. Three-frame translation of transcripts is a middle ground. Here each annotated or assembled transcript is translated in its three forward frames, so that untranslated regions, retained introns and alternative frames are included, but only for sequences known to be transcribed. This yields a database much smaller than a six-frame genome translation but still much larger than the proteome, and like the six-frame approach it is dominated by sequences that are not translated. Variant databases Most proteogenomic studies of human samples add sequence variants. The inputs are variant calls in a standard format, derived from whole-genome, whole-exome or RNA sequencing of the sample, or from public catalogues of population variation such as dbSNP and gnomAD, or of cancer mutations such as COSMIC. A tool then annotates each variant against the reference gene models, determines its effect on the coding sequence and writes out the altered protein sequences. The details matter more than they might seem. A missense variant changes one residue, and the affected tryptic peptide is the unit that matters, so many tools write out just that peptide, with flanking context, rather than duplicating the whole protein. A variant that creates or destroys a lysine or arginine changes where trypsin cuts, altering not one peptide but the boundaries of two. Insertions, deletions and frameshifts must be handled by translating the altered coding sequence through to the next stop codon. Variants that lie close together on the same haplotype should in principle be applied together, since a peptide carrying both exists only if they are on the same chromosome. And a variant falling in multiple transcripts of the same gene may produce different peptides in each. Several well-established tools perform these steps. customProDB, an R package described by Wang and Zhang in 2013, generates customised databases from RNA-seq data, including variants, novel junctions and a filter that keeps only transcripts expressed above a threshold in the sample. PGA, a Bioconductor toolkit from the same group, extends this to include novel transcripts and to handle the downstream search and error control. Other pipelines exist in Galaxy-based platforms, in Python packages that operate on variant call files, and within larger frameworks for neoantigen discovery. A basic question is which variants to include. Adding all known population variants, many millions of them, inflates the database with sequences mostly absent from any one sample. Sample-specific variants are far fewer, but their inclusion depends on having good sequencing data and on the variant caller's error rate. A reasonable general rule is to include the variants called in the sample itself, with quality filters, and to treat population catalogue variants as a separate, lower-prior category if they are used at all. Splice junctions and assembled transcripts RNA sequencing reads that span exon-exon junctions provide direct evidence of which splice events occurred. After aligning reads with a splice-aware aligner such as STAR or HISAT2, junctions supported by a minimum number of reads, and ideally not previously annotated, can be translated into short peptide sequences covering the junction. Sheynkman and colleagues showed in 2013 that searching such junction peptides allows novel splice events to be confirmed at the protein level, though with modest yield. Because each junction contributes only one or a few peptides, the database grows little, and the added sequences are specific to what was observed in the sample. The reading frame is the subtle part. A novel junction between two annotated coding exons can be translated in the frame of the upstream exon, and the result is unambiguous if the downstream frame is preserved. If the junction shifts the frame, or joins exons not normally connected, the translation beyond the junction may be a novel sequence. Tools typically generate all frames consistent with an upstream open reading frame, or, more conservatively, only the frame of the annotated upstream coding sequence. Full transcript assembly, with tools such as StringTie or Trinity, goes further. It reconstructs entire transcripts from the reads, including isoforms that combine known exons in new ways, transcripts from unannotated loci and fusion transcripts. The assembled transcripts must then be translated, which requires choosing reading frames. Programs such as TransDecoder predict the most likely coding region of each transcript, usually based on length and sequence composition. That choice is a hypothesis in its own right: a transcript may contain several plausible reading frames, or none that is actually translated. Assembly also produces errors, particularly for lowly expressed transcripts and in repetitive regions, and chimeric assemblies can generate peptide sequences that exist nowhere in the cell. Assembled databases are therefore powerful for discovering isoforms and fusions, and they require the same scepticism as any other computational prediction. Fusion genes deserve special mention. Dedicated fusion callers, such as STAR-Fusion or Arriba, identify chimeric transcripts from RNA-seq with higher specificity than general assembly. The peptide spanning the fusion junction, where it maintains or shifts the reading frame of the downstream partner, is a sequence unique to the tumour, and in some cancers, such as those driven by recurrent fusions, a highly attractive target. Ribosome profiling Ribosome profiling, introduced by Ingolia and colleagues in 2009, sequences the fragments of messenger RNA protected by translating ribosomes from nuclease digestion. These footprints, around 28 to 30 nucleotides long, map to the positions ribosomes occupied at the moment of harvest. Because ribosomes move three nucleotides at a time, the footprints show a characteristic three-nucleotide periodicity in actively translated regions, and this periodicity can be used to infer which reading frame is being translated. Treatment with drugs that stall ribosomes at initiation sites, such as harringtonine or lactimidomycin, further allows start codons to be mapped. Algorithms such as RiboTaper, ORF-RATER, RibORF and PRICE use these signals to call translated open reading frames across the transcriptome. Their output is exactly what a proteogenomic database wants: a list of reading frames with direct evidence of translation, including many that annotation ignores, such as upstream open reading frames, short reading frames in long non-coding RNAs and reading frames overlapping known genes in alternative frames. A Ribo-seq-derived database is much smaller and much more specific than a three-frame translation, because it contains only frames that ribosomes were seen to engage. Ribosome profiling is not a direct measurement of protein, however. Ribosome occupancy shows that translation began, not that a stable product accumulated. Many of the products of short reading frames are expected to be rapidly degraded, and some may never be detectable by standard shotgun proteomics even when translation is real. Matched Ribo-seq and mass spectrometry data from the same samples is valuable for this reason: it allows a translated frame to be recorded as translated, and its product, if found, to be recorded as detected, without conflating the two. Assembly and housekeeping The final database is assembled from the reference proteome plus the chosen additions, and then cleaned. Several housekeeping steps matter a great deal. Redundancy should be removed. Many novel sequences share most of their residues with reference proteins; a variant protein differs by one residue, a junction isoform by a few. Including full redundant sequences inflates the database and complicates protein inference without adding identifiable peptides. Many pipelines therefore write out only the novel peptides, or short sequences around the novel region, and remove any novel peptide whose sequence already exists elsewhere in the reference, including after treating leucine and isoleucine as equivalent, since mass spectrometry cannot distinguish them. Every entry should carry provenance. The sequence header should record which source generated it, which gene and transcript it derives from, which variant or junction it covers, and what supporting evidence existed, for instance the number of RNA reads or the Ribo-seq score. Without provenance, the novel identifications at the end cannot be traced back or stratified, and the separate error control recommended in the next chapter becomes impossible. Common contaminants should be included: keratins, trypsin, serum albumin and other proteins routinely introduced during sample handling. A contaminant spectrum with no correct target may otherwise be matched to a novel sequence. Decoys should be generated from the final combined database, not from the reference alone, and with the same method for all parts. If decoys are made only from the reference, the novel sequences have no decoy counterparts and their false matches go uncounted. A practical design sequence In practice, a careful database design for a human proteogenomic study might proceed as follows. Start from a well-curated reference proteome, such as the GENCODE basic set or UniProt reviewed human proteins with isoforms, plus contaminants. Add sample-specific variants called from matched DNA or RNA sequencing data, filtered for quality and, for RNA-based calls, for adequate read support. Add novel junction peptides supported by several RNA reads. If fusions matter, add junction peptides from a dedicated fusion caller. If non-canonical translation is the question and Ribo-seq data exist, add Ribo-seq-supported reading frames; if not, add a restricted three-frame translation of expressed transcripts, recognising the cost. Remove redundant sequences, record provenance, generate decoys from the whole, and keep the categories labelled so that they can be evaluated separately after the search. The principle behind this sequence is to include sequences in proportion to the evidence that they might be present. Each addition should be justified by data from the sample or by a specific question. Everything included is a candidate for identification, but also a candidate for a false match, and the next chapter explains why that second fact becomes more pressing as the database grows. Hashtags: #Proteogenomics #Genomics #Transcriptomics #MassSpectrometry #ShotgunProteomics #CustomProteinDatabases #SampleSpecificDatabases #VariantPeptides #SomaticMutations #AlternativeSplicing #SpliceJunctionPeptides #NonCanonicalTranslation #RibosomeProfiling #NeoantigenDiscovery #Immunopeptidomics #PeptideSpectrumMatching #TargetDecoyStrategy #FalseDiscoveryRate #ProteinInference #SpectralValidation #SyntheticPeptideValidation #RetentionTimePrediction #CPTAC #PersonalizedProteomics #FutureOfProteogenomics

Download the Book (PDF): Introduction In the late 1950s, John Kendrew and his colleagues in Cambridge built a model of sperm whale myoglobin out of rods and clips, and the thing that struck everyone who saw it was how untidy it looked. Chemists had expected a protein to be orderly in the way a salt crystal is orderly, perhaps a neat stack of repeating units. Instead the molecule was a knotted, asymmetric tangle of helices wrapped around a heme group, with no obvious rule governing where each segment went. The shock was productive. It told biologists that the function of a protein lived in the exact placement of its atoms, and that the only way to know that placement was to see it. For more than six decades since, the most reliable way of seeing it has been to coax the protein into a crystal, shine X-rays through it, and work backwards from the pattern of scattered spots to a map of electron density. This book is about that working backwards, and about everything that has to go right before it can begin. It follows a single path from a purified protein in a tube to an interpretable electron density map, stopping at the four stations where most projects succeed or fail: crystallization by vapor diffusion and its many refinements; the harvesting and cryo-cooling of crystals; the collection of diffraction data at synchrotron beamlines; and the determination of phases, whether by borrowing them from a known structure through molecular replacement or by measuring them experimentally through anomalous scattering in MAD and SAD experiments. Why this chain still matters It is fair to ask why anyone should learn crystallography in detail in 2026. Two developments of the past decade might seem to have made it a legacy craft. The first is the resolution revolution in single-particle electron cryomicroscopy, which since roughly 2013 has allowed large complexes, membrane proteins and flexible assemblies to be solved without ever forming a crystal. The second is AlphaFold2, released by DeepMind in 2021, and its successors, which predict the folds of single chains with an accuracy that often rivals experiment. The 2024 Nobel Prize in Chemistry, shared by Demis Hassabis and John Jumper for structure prediction and by David Baker for computational protein design, formalised the sense that the field had changed. Yet crystallography did not shrink into irrelevance. It changed its role. Prediction has made molecular replacement easier than it has ever been, so that many structures which would once have required a heavy-atom derivative or a selenomethionine preparation can now be phased from a computed model in an afternoon. At the same time, the questions that remain hard are exactly those where crystallography's particular strengths count: the precise position of a bound drug fragment, the protonation state implied by a hydrogen-bonding network, the ordered water in an active site, the subtle conformational change between two ligand states, the atomic detail of small proteins too little for electron microscopy to image well. Pharmaceutical fragment campaigns routinely soak hundreds or thousands of crystals of a single target and collect data on each, a throughput that no other structural method yet matches. The Protein Data Bank passed 200,000 entries in early 2023, and the large majority of its entries still come from X-ray diffraction. What has changed is who does the work and how. A graduate student in the 1990s might spend months growing crystals, days at a rotating-anode generator, and weeks interpreting heavy-atom Patterson maps. Today the same student may set up trays with a nanolitre robot, ship frozen crystals in a puck to a synchrotron hundreds of kilometres away, watch data being collected remotely through a web browser, and receive automatically processed, sometimes automatically phased, results before the next morning. The automation is extraordinary, and it has a cost. When every step is a button, it becomes possible to finish a structure without understanding why it worked, and therefore impossible to know what to do when it does not. The student who cannot tell a twinned crystal from a poorly processed one, or a genuine anomalous signal from noise, or a cryoprotectant problem from a radiation damage problem, is stuck the moment the pipeline returns an error. The argument of this book The controlling idea of what follows is simple to state. A crystal structure is the end of a long chain of decisions, and information is lost at every link. A protein that is slightly heterogeneous crystallizes poorly. A crystal that is poorly cryoprotected diffracts to lower resolution than it could have. A dataset collected with too high a dose carries the fingerprints of radiation damage into the anomalous differences that were supposed to provide the phases. A molecular replacement search model with the wrong domain arrangement produces a solution that looks plausible and refines badly. The phase problem, the famous obstacle at the heart of crystallography, is solved in practice not by one clever algorithm but by the accumulated quality of every upstream choice. The practitioner who understands where information leaks away is the one who can stop the leak. This framing explains the book's structure. The first chapter sets out the physics of diffraction and explains precisely what the phase problem is, because every later decision is easier to understand once one knows what the experiment measures and what it cannot. The next three chapters deal with the sample: preparing a protein that is fit to crystallize, understanding vapor diffusion as a controlled walk across a phase diagram, and optimizing the first poor hits into crystals worth taking to a beamline. The fifth chapter treats harvesting and cryo-protection, a step that is often rushed and that destroys a surprising fraction of good crystals. The sixth describes synchrotron sources and beamlines, and how an experimenter should think about dose, wavelength and strategy. The seventh follows the images through indexing, integration and scaling, and explains how to read the statistics that come out. The last two chapters address the two families of phasing methods: molecular replacement, now transformed by predicted models, and experimental phasing through anomalous scattering, from the multiwavelength experiments of the late 1980s to the native sulfur SAD experiments that long-wavelength beamlines have made practical. How to read it The intended reader is a scientist from a neighbouring field, a cell biologist who needs a structure of her favourite enzyme, a chemist designing inhibitors, a computational biologist who uses crystal structures daily and wants to know how much to trust them, or a student starting in a structural biology group. No prior training in crystallography is assumed, but some willingness to follow a physical argument is. Mathematics appears only where it clarifies; where an equation would serve, it is described in words. A second kind of reader may also find the book useful: the crystallographer who learned the trade after automation arrived and who wants to understand what the software is doing on their behalf. Much of the tacit knowledge of the field, why one adds glycerol slowly, why one collects a low-dose pass first, why an anomalous signal should be judged by its correlation between half-datasets rather than by eye, was once passed on at the bench. It is set down here in plain prose. Two limits of scope should be named. The book concentrates on macromolecular crystallography with conventional synchrotron sources, and mentions serial methods at X-ray free-electron lasers and microfocus beamlines only where they illuminate the main path. It also stops at the point where a phased, interpretable map exists, touching on model building, refinement and validation only enough to explain how the quality of phases is judged. Those later stages matter enormously, but they are a subject of their own, and the decisions that most often doom a project are made well before them. Finally, a note on tone. Crystallography has a romantic history, full of heroic figures and long campaigns: Perutz's twenty-two years on haemoglobin, Hodgkin's decades on insulin and vitamin B12, the ribosome structures that earned the 2009 Nobel Prize. The romance is deserved, but it can mislead. Most successful structures today come not from heroic persistence but from systematic, well-informed choices made early. The aim here is to make those choices visible. Chapter 1: What Diffraction Measures and What It Hides A protein molecule is a few nanometres across and contains thousands of atoms separated by distances of about 1.5 ångströms, a tenth of a nanometre and a half. To resolve features on that scale one needs radiation whose wavelength is comparable, and X-rays in the range of roughly 0.7 to 2 ångströms fit. But there is no lens that can focus X-rays scattered by a molecule back into an image in the way a light microscope does. The scattered waves can be recorded, but not recombined. Everything distinctive about crystallography, from the need for crystals to the phase problem, follows from that single limitation. Why a crystal, and not a single molecule A single protein molecule scatters X-rays so weakly that the signal would be lost in background long before enough photons were collected, and the molecule would be destroyed by radiation in the attempt. A crystal solves this by arranging an enormous number of identical molecules, typically in the range of 10 to the power of 12 or more for a crystal a few tens of micrometres across, in a regular three-dimensional lattice. The waves scattered by each molecule add together. In most directions they arrive out of step and cancel. In a small set of special directions they arrive in step and reinforce one another, producing strong discrete beams. Those beams are the spots, or reflections, recorded on the detector. The geometry of those special directions is governed by the lattice, not by the molecule. William Lawrence Bragg's formulation, from 1912, treats the crystal as a stack of parallel planes passing through lattice points. Reinforcement occurs when the path difference between waves reflected from successive planes equals a whole number of wavelengths. Planes that are closely spaced give reflections at high scattering angles; widely spaced planes give reflections near the direct beam. Each reflection is labelled by three integers, the Miller indices h, k and l, which identify the family of planes that produced it. A protein dataset may contain tens of thousands to several hundred thousand unique reflections. The spacing of the finest planes whose reflections can still be measured defines the resolution of the experiment. A dataset at 3 ångströms resolution contains reflections from planes as close as 3 ångströms apart, and the resulting electron density map shows the course of the polypeptide chain and most large side chains, but little atomic detail. At 2 ångströms, individual side chains and many ordered water molecules become clear. At 1.2 ångströms and better, individual atoms separate, and one begins to see alternative conformations and, with luck, some hydrogen atoms. Resolution in crystallography is therefore not a property of the instrument, as it is in microscopy, but chiefly a property of the crystal: how well ordered the molecules are from one unit cell to the next. The crystal repeats in three dimensions a box called the unit cell, whose edges and angles are the cell parameters. Inside the unit cell, symmetry operations such as rotations and screw axes may relate several copies of the molecule; the smallest unique portion from which the rest can be generated is the asymmetric unit. Because proteins are built from L-amino acids and are therefore chiral, they can crystallize only in the 65 space groups that lack mirror planes and inversion centres. The space group determines which reflections are systematically absent, which are related by symmetry, and how many unique measurements are needed for a complete dataset. Protein crystals differ from salt or small-molecule crystals in one respect that shapes the whole field: they are wet. Typically between 30 and 70 percent of the crystal volume is solvent, filling channels between the molecules. Brian Matthews showed in 1968 that across the protein crystals then known, the volume of the unit cell per dalton of protein fell within a characteristic range, most often around 2 to 3 cubic ångströms per dalton, corresponding to roughly 40 to 60 percent solvent. That empirical relation, the Matthews coefficient, remains the first tool for guessing how many molecules sit in the asymmetric unit. The high solvent content is why protein crystals are soft, fragile and sensitive to drying; why small molecules can be soaked into them; and why cooling them without forming ice becomes a problem in its own right. Structure factors, amplitudes and phases Each reflection corresponds to a wave with two properties: an amplitude, which describes how strong it is, and a phase, which describes where its peaks fall relative to an arbitrary origin in the unit cell. Crystallographers bundle these into a complex number called the structure factor. The structure factor for a reflection is the sum of the waves scattered by every atom in the unit cell, each weighted by how strongly that atom scatters and shifted in phase according to the atom's position. The electron density at every point in the unit cell can be computed from the full set of structure factors by a Fourier synthesis: add up waves, one per reflection, each with the correct amplitude and the correct phase, and the peaks of the sum fall on the atoms. This is the calculation a lens would perform physically if X-ray lenses existed. Crystallographers perform it numerically. The difficulty is that detectors count photons, and the number of photons in a spot is proportional to the square of the amplitude. The phase leaves no trace in the intensity. After a diffraction experiment, the crystallographer knows how strong each of the constituent waves was, but not how they were aligned. This is the phase problem, and it is not a technical inconvenience that better detectors might one day remove. It is a consequence of measuring intensities of waves oscillating at around 10 to the power of 18 times per second. How much information is lost? More than intuition suggests. A well-known demonstration, often attributed in teaching to the cat-and-duck illustrations popularised by Kevin Cowtan, combines the Fourier amplitudes of one picture with the phases of another. The resulting image resembles the picture that contributed the phases, not the one that contributed the amplitudes. Phases carry most of the structural information. A map computed with correct amplitudes and random phases is noise; a map computed with approximately correct phases and even crude amplitudes is recognisable. This asymmetry explains why so much of crystallography is devoted to obtaining phases, and why a biased phase estimate, such as one borrowed from a wrong model, can produce a map that shows what the experimenter expected rather than what is there. What intensities alone can tell you Although phases are lost, the intensities are not uninformative. Several things can be learned from them directly, and these guide the choices discussed in later chapters. The first is the Patterson function, introduced by Lindo Patterson in 1934. It is a Fourier synthesis using the squared amplitudes and no phases at all. The resulting map does not show atoms; it shows vectors between atoms. Every peak in a Patterson map corresponds to a pair of atoms, located at the position given by the vector from one to the other. For a protein with thousands of atoms the Patterson map is hopelessly crowded. But for a small number of heavy atoms whose scattering dominates a difference signal, the Patterson map is tractable, and for decades it was the way heavy-atom positions were found. It also underlies the rotation and translation searches of molecular replacement, because the self-vectors within a molecule depend on its orientation but not its position. The second is the Wilson plot, from Arthur Wilson's statistical analysis of 1949. If the atoms are distributed roughly at random, the average intensity in a shell of resolution falls off in a predictable way that depends on the overall atomic displacement, the so-called B-factor, which reflects thermal vibration and static disorder. A Wilson plot places the data on an approximately absolute scale and gives a first estimate of how well ordered the crystal is. Unusual shapes in it can flag problems such as ice rings or anisotropy. The third is intensity statistics. The distribution of intensities within a resolution shell follows characteristic forms for ordinary crystals. Crystals that are twinned, meaning they contain domains in different orientations whose diffraction patterns overlap, show abnormal distributions, with too few very weak and very strong reflections. Tests based on these statistics, such as the L-test of Padilla and Yeates, are now routinely applied by data-processing programs and warn the experimenter before any time is spent on phasing. The fourth, and the most important for phasing, is the small breakdown of symmetry caused by anomalous scattering. Under ordinary conditions a reflection and its inverse, the pair indexed h, k, l and minus h, minus k, minus l, have equal intensity; this is Friedel's law. When the X-ray energy is close to an absorption edge of an atom in the crystal, that atom scatters with a phase shift that breaks Friedel's law. The two members of the pair, called a Bijvoet pair after Johannes Bijvoet, who used the effect in 1951 to determine the absolute configuration of a chiral molecule, then differ slightly in intensity. The differences are small, often a few percent or less, but they are measurable, and they encode information about the positions of the anomalous scatterers. The whole of MAD and SAD phasing rests on this effect. The routes around the phase problem There are three broad strategies for recovering phases, and each is the subject of later chapters. It helps to have the shape of all three in mind from the start. The first is to borrow them. If a structure of a similar molecule is already known, or can be predicted, it can be placed in the unit cell of the new crystal by searching for the orientation and position that best reproduce the observed intensities. Phases calculated from the placed model are then combined with the observed amplitudes to give a first map. This is molecular replacement, proposed by Michael Rossmann and David Blow in 1962. It is now the method behind most new crystal structures, a proportion that has grown sharply since predicted models became widely available. The second is to perturb the crystal in a controlled way and measure the effect. In isomorphous replacement, heavy atoms such as mercury, platinum or gold are soaked into the crystal without changing its structure otherwise. The differences in intensity between native and derivative data reveal the heavy-atom positions, and from these, phases can be estimated. Max Perutz and colleagues demonstrated the approach on haemoglobin in 1954, and it produced the first protein structures, myoglobin in 1958 and haemoglobin in 1959. In anomalous methods, the perturbation is a change of X-ray energy rather than a change of crystal: by measuring at energies near an element's absorption edge, the experimenter makes that element scatter differently and extracts phase information from the resulting Bijvoet differences. Multi-wavelength anomalous diffraction (MAD) uses several energies; single-wavelength anomalous diffraction (SAD) uses one, supplemented by computational density modification. The third is to calculate phases directly from the intensities using statistical relationships among them. These direct methods, for which Herbert Hauptman and Jerome Karle shared the 1985 Nobel Prize in Chemistry, transformed small-molecule crystallography. They work because, for structures made of discrete atoms and measured at atomic resolution, the phases of strong reflections are not independent of one another. For proteins, true ab initio direct methods succeed only rarely, when data extend to about 1.2 ångströms or better and the protein is small. But direct methods found an essential role in locating the substructure of heavy or anomalous atoms, where the problem resembles a small-molecule problem. Programs such as SHELXD apply them to find dozens of selenium sites from anomalous differences alone. Alongside all three routes runs a fourth idea that is less a phasing method than a way of improving phases: density modification. A true electron density map of a protein crystal has known general properties. The solvent regions are flat; the protein regions have a characteristic distribution of density values; if several copies of the molecule sit in the asymmetric unit, their density should be identical. Imposing these expectations on a noisy map and recalculating phases improves them, often dramatically. Solvent flattening, histogram matching and non-crystallographic symmetry averaging, developed from the 1970s onwards by Bi-Cheng Wang, Kam Zhang, Kevin Cowtan and others, are what make single-wavelength methods feasible. The higher the solvent content and the more copies in the asymmetric unit, the more powerful density modification becomes. Why phase quality depends on everything upstream It would be convenient if the phase problem could be treated separately from the rest of the experiment, as a computational puzzle to be solved once the data are in hand. It cannot. Each route around it has specific demands on the data, and those demands reach back through the beamline to the crystal and the protein. Molecular replacement requires data complete enough, especially at low resolution, that the search functions are not distorted; it fails more often when data are incomplete or when strong low-resolution reflections are missing behind the beamstop or lost to overloads. Anomalous phasing requires data measured with high redundancy and accuracy, because the signal it relies upon is a difference of a few percent between two large numbers. That, in turn, requires crystals that tolerate enough dose to measure every reflection many times, which requires good cryo-protection, which depends on crystals that were grown well in the first place. Isomorphous replacement requires that soaking a heavy atom leaves the crystal's cell and packing unchanged, which depends on how robust the crystal contacts are. Density modification depends on solvent content, which depends on which crystal form grew. Consider a concrete case. A crystal of a selenomethionine-labelled protein is harvested into a cryoprotectant that is slightly too weak. A thin ring of ice forms as the crystal is plunged. The data processing software excludes the resolution shell containing the ice ring, so that shell is incomplete. The ice also raises background across the pattern, reducing the precision of every weak measurement. The experimenter, worried about the diffraction limit, increases exposure, which accelerates radiation damage, and selenomethionine is among the residues whose side chains are damaged early. The anomalous signal therefore decays over the course of the dataset. The substructure search finds only half the selenium sites, the phases are poor, and the map is uninterpretable. The problem appears at phasing, but its origin lies in a harvesting decision made days before. This chain is the reason the rest of the book starts with the protein rather than the phasing algorithms. Crystallographers sometimes say that there is no substitute for good crystals, and it is true in a precise sense: every quantity that later determines whether phases can be found, from resolution and completeness to anomalous signal and solvent content, is fixed or bounded by the sample. A note on what a map is One last conceptual point belongs here because it frames the discussion of phasing at the end of the book. The output of a crystallographic experiment is not an atomic model. It is an electron density map, averaged over every molecule in the crystal and over the duration of the measurement. The atomic model is an interpretation of that map, constrained by chemical knowledge of bond lengths, angles and planarity. Where the map is clear, the model is a faithful summary; where the map is weak, perhaps because a loop is flexible or a ligand is partially occupied, the model is a hypothesis. The quality of the map depends on two things: the accuracy of the amplitudes, which is a question of data collection, and the accuracy of the phases, which is a question of phasing and refinement. Crystallographers express phase accuracy as a figure of merit, which ranges from zero for random phases to one for perfect phases, and which estimates the cosine of the average phase error. Maps with a figure of merit around 0.3 are usually hard to interpret after initial experimental phasing; density modification commonly raises that substantially, and a figure of merit in the region of 0.6 to 0.8 generally gives a map in which the chain can be traced. These are rules of thumb, not thresholds, and the practical test is always whether the map shows features one did not put there: side chains with the right shapes, carbonyl bumps along the backbone, the flat outline of a ring. That test, the appearance of unexpected detail, is also the best defence against the most insidious failure of crystallography, model bias, in which phases borrowed from a model reproduce the model in the map. It will return in the chapter on molecular replacement. For now, the lesson is that crystallography's central problem is not simply that phases are missing, but that the ways of estimating them each carry their own risks of error, and that those risks are smallest when the data and crystals are best. The next three chapters turn to how good crystals are made. Chapter 2: A Protein Fit to Crystallize Most crystallization failures are decided before a single drop is set up. A protein that is aggregated, partly unfolded, proteolytically nibbled, heterogeneously modified or floppy at its termini can be screened against thousands of conditions and never form a crystal, because the thing in the tube is not one molecule but a population of slightly different ones. Crystals are exacting. They require that the same surface patches of the same molecule meet in the same way, millions of times over in every direction. Anything that makes molecules differ from one another works against that requirement. Structural genomics consortia, which in the 2000s attempted to solve structures of thousands of proteins in parallel, produced the clearest evidence of how steep the attrition is. Their public progress records, tracked through the TargetTrack database and its predecessors, showed that only a minority of cloned targets yielded purified protein, a minority of those yielded crystals, and a minority of crystals yielded structures. The exact percentages varied by consortium and organism, but the pattern was consistent, and the largest single losses occurred at the stages of soluble expression and crystallization. Those projects also generated much of the systematic understanding of how to improve the odds, and this chapter draws on it. Constructs: deciding what to crystallize The first decision is which piece of the protein to express. Full-length proteins often carry disordered regions, flexible linkers, signal peptides or membrane anchors that are functionally important but hostile to crystallization. A disordered tail of thirty residues does not show up in the final electron density, but its presence in the tube can prevent the ordered part from packing into a lattice at all. The practical response is to design several constructs with different boundaries and test them in parallel. Boundaries are chosen from sequence alignments, where conserved regions often mark folded domains; from secondary structure and disorder predictions; from limited proteolysis, in which a small amount of protease trims flexible regions and mass spectrometry identifies the stable core that remains; and, since 2021, from predicted structures. AlphaFold models come with a per-residue confidence score, and regions of very low confidence correspond well to regions that are disordered in solution. Trimming such regions from a construct, or at least testing constructs with and without them, has become a routine first step. A predicted model also shows which domains form a compact unit and which are tethered by long linkers, which helps decide whether to crystallize domains separately. Expression tags require similar thought. A polyhistidine tag makes purification by nickel affinity easy, but it is flexible and can hinder packing. Many laboratories include a protease site, such as that for tobacco etch virus protease or the 3C protease from human rhinovirus, so that the tag can be removed before crystallization. Others screen both tagged and cleaved forms, since tags occasionally help by forming crystal contacts. Surface engineering and crystallization chaperones When a well-defined domain still refuses to crystallize, the problem is often its surface. Crystal contacts are formed by patches of the protein surface that happen to complement each other. Surfaces rich in long, flexible, charged side chains, particularly lysine and glutamate, pay a large entropic cost to become immobilised in a contact. Zygmunt Derewenda and colleagues developed the surface entropy reduction strategy, published in the early 2000s, in which clusters of such residues are mutated to alanine. The mutations reduce the entropy penalty and create patches more likely to form contacts. A web server, SERp, was built to suggest candidate clusters. Numerous structures have been solved this way, and the approach is now a standard option when wild-type protein fails. Other strategies add a partner that brings its own good crystallization behaviour. Fusion proteins, such as maltose-binding protein attached through a short rigid linker, or T4 lysozyme inserted into a loop, have been used widely; the latter played an essential part in the first structures of G-protein-coupled receptors such as the beta-2 adrenergic receptor in 2007. Antibody fragments, including Fab fragments and camelid single-domain antibodies (nanobodies), bind to a specific conformation, rigidify flexible regions and provide large, well-behaved surfaces for packing. Designed binding proteins such as DARPins serve similar purposes. Each of these chaperones can make an intractable target crystallizable, at the price of complicating the construct and sometimes altering the conformation one wished to observe. Chemical modification is a simpler lever. Reductive methylation of lysine residues, which converts primary amines to dimethylamines, changes surface properties without changing charge and has rescued crystallization for a meaningful fraction of recalcitrant proteins in structural genomics pipelines. Removal or trimming of glycans, by expressing in cell lines that produce simpler glycosylation or by enzymatic treatment with endoglycosidases, reduces the heterogeneity that glycosylated proteins bring. Purity, homogeneity and stability Purity is necessary but not sufficient. A protein that runs as a single band on an SDS gel may still be a mixture of monomers and dimers, partially unfolded species, or molecules with different ligands bound. What matters for crystallization is homogeneity: that the molecules in solution are identical in conformation and oligomeric state. Several measurements help establish homogeneity. Size-exclusion chromatography, which should show a single symmetric peak at the expected elution volume, is the first check and also the last purification step in most protocols. Multi-angle light scattering coupled to size exclusion gives absolute molecular mass and reveals whether the peak is a single species. Dynamic light scattering measures the distribution of particle sizes in the final sample and is sensitive to small amounts of aggregate, which can poison nucleation. Mass spectrometry of the intact protein confirms that the mass matches the construct and reveals truncations, unexpected modifications or incomplete incorporation of selenomethionine when that is intended. Stability can be probed by differential scanning fluorimetry, often called a thermal shift assay. A dye that fluoresces when bound to exposed hydrophobic surface is mixed with the protein, the temperature is ramped, and the midpoint of the unfolding transition is recorded. Running the assay across a panel of buffers, pH values, salts and additives identifies conditions in which the protein is most stable. Ligands that bind specifically usually raise the melting temperature, which makes the assay useful both for choosing a buffer and for choosing a ligand to co-crystallize. There is reasonable evidence from structural genomics analyses that more stable proteins crystallize more readily, which is intuitive: a protein that is partly unfolding at room temperature presents a moving target for lattice formation. The relationship is a tendency, not a law, and many successful projects began with modest stability. Ligands deserve special mention. Enzymes crystallized in the presence of a substrate analogue, cofactor or inhibitor often crystallize better than the apoenzyme, because the ligand locks a single conformation. Adding a known binder is one of the most effective and least expensive things to try when a protein will not crystallize. Concentration, buffer and the final sample The final sample for crystallization is typically concentrated to somewhere between 5 and 30 milligrams per millilitre, although the right value varies widely with molecular weight and solubility. Too little protein and nothing nucleates; too much and every drop precipitates. A useful practice is to check the protein's behaviour against a couple of standard precipitant conditions before committing to a full screen. If most drops in a screen show heavy precipitate, the protein is probably too concentrated; if most remain clear, too dilute. The buffer in which the protein sits should be simple. Every component will be diluted into the crystallization drop and will influence what happens there. Phosphate buffers can precipitate with divalent cations in screens and produce salt crystals that fool the eye. High glycerol concentrations, used to stabilise proteins in storage, alter the vapor diffusion equilibrium and can inhibit crystallization. Reducing agents such as dithiothreitol oxidise over time; tris(2-carboxyethyl)phosphine is more stable, though it also has its quirks. The common advice is to use a low concentration of a buffer chosen for the protein's stable pH range, enough salt to keep it soluble, a reducing agent if there are free cysteines, and nothing else unless there is evidence it helps. The sample should be cleared of dust and aggregates by centrifugation or filtration immediately before setting up trays. Aggregates act as heterogeneous nucleation sites and can produce showers of tiny crystals or amorphous precipitate. Freshness matters too. A protein that is stable for a week at 4 degrees Celsius may still crystallize better on day one than day six, and many laboratories flash-freeze small aliquots so that every experiment can start from the same state. Selenomethionine and other labels for phasing If experimental phasing may be needed, the sample preparation stage is the moment to plan for it. The most common approach, established by Wayne Hendrickson and colleagues in a 1990 paper that used selenomethionyl ribonuclease H, is to replace every methionine in the protein with selenomethionine, in which the sulfur atom is replaced by selenium. Selenium has an absorption K-edge at about 12.66 kiloelectronvolts, corresponding to a wavelength near 0.98 ångströms, which sits comfortably within the tunable range of most synchrotron beamlines. Selenomethionine is usually well tolerated, and selenomethionyl proteins generally crystallize under the same conditions as the native. In Escherichia coli, labelling is achieved either by using a methionine-auxotrophic strain grown in medium supplemented with selenomethionine, or by the metabolic inhibition approach, in which a mixture of amino acids suppresses methionine biosynthesis in a normal strain before selenomethionine is added. Incorporation is typically high, and mass spectrometry confirms it by the characteristic mass shift of about 47 daltons per substituted methionine. Labelling in eukaryotic expression systems such as insect or mammalian cells is possible but harder, because selenomethionine is more toxic to those cells and incorporation is often incomplete. A rough guide, drawn from the early MAD literature and still widely quoted, is that one well-ordered selenium site is enough to phase somewhere in the region of 75 to 100 residues, though successful cases span a wide range depending on data quality. If the protein has few methionines, some groups introduce extra ones by mutating leucine or isoleucine residues in the hydrophobic core, which are chemically similar. Selenium incorporated at the flexible amino-terminal methionine contributes little, because a disordered site gives no coherent anomalous signal. For proteins where selenomethionine is impractical, alternatives exist. Native sulfur atoms in cysteine and methionine give a weak anomalous signal at longer wavelengths, which the chapter on experimental phasing discusses. Halide ions such as bromide and iodide can be introduced by brief soaks of crystals, a method developed by Zbigniew Dauter and colleagues around 2000, and bind to the protein surface in many ordered sites. Lanthanide complexes and other heavy-atom compounds can be soaked or co-crystallized. The choice is best made early, because some of these routes require a different protein preparation. Membrane proteins as a special case Integral membrane proteins present every difficulty discussed above in an amplified form. They must be extracted from membranes with detergents, which form micelles around the hydrophobic surface; the detergent-protein complex is larger, less homogeneous and less stable than a soluble protein; and the hydrophilic surface available for crystal contacts is small. Detergent choice becomes a critical variable: detergents with short chains give smaller micelles that allow more protein-protein contact but tend to destabilise the protein, while longer-chain detergents stabilise but hinder packing. The detergent lauryl maltose neopentyl glycol, introduced around 2010, became popular because it stabilises many membrane proteins well. Thermostabilising mutations, identified by systematic scanning mutagenesis, were a key ingredient in many GPCR structures of the 2010s, and the company Heptares, founded on this approach, built a drug discovery platform around it. Crystallization chaperones such as antibody fragments and fusion proteins are especially valuable for membrane proteins because they add hydrophilic surface. And, as the next chapter discusses, crystallization in lipidic cubic phase offers an environment closer to the membrane and has produced many of the highest-resolution membrane protein structures. It is worth stating directly that for many large membrane protein complexes, the preferred route in 2026 is single-particle electron cryomicroscopy rather than crystallography. Crystallography remains competitive for small membrane proteins, for high-resolution studies of ligand binding, and for fragment screening, where throughput matters. But the decision about which method to use is now part of sample preparation, and should be made with the target's size, stability and flexibility in mind. A worked example of sample triage It helps to see how these tools combine in practice. Imagine, as a composite of many real projects, a bacterial two-domain enzyme of about 45 kilodaltons that purifies well, looks clean on a gel, and produces nothing but light precipitate across two commercial screens. The temptation is to buy a third screen. The better move is to ask what the protein is doing in the tube. Size-exclusion chromatography with light scattering shows a main peak at the mass of a monomer but a shoulder at roughly twice that mass, and dynamic light scattering reports a polydisperse sample. Intact mass spectrometry reveals two species differing by the mass of the first eleven residues, indicating that a protease in the expression host clips the amino terminus in some molecules but not others. The predicted structure shows those eleven residues, and a further stretch of about twenty residues at the carboxyl terminus, as low-confidence regions with no contacts to the folded domains. A thermal shift screen shows that the protein's melting temperature rises by several degrees in a buffer at slightly higher pH with a moderate salt concentration, and rises further when a known product analogue is added. Each observation suggests a specific change. A new construct lacking both flexible termini removes the source of heterogeneous clipping and the floppy tail. The improved buffer and the ligand address stability. Running size exclusion in the new buffer shows a single symmetric peak; the dimer shoulder, which was probably a nonspecific aggregate of partly unfolded molecules, is gone. With nothing else changed, the first screen of the new sample gives several hits. None of these steps is sophisticated. What made them effective was measuring the sample before blaming the screen. The same logic applies in reverse: a sample that passes every biophysical test yet still fails to crystallize is a strong candidate for surface engineering, chemical methylation or a chaperone, because the problem has been narrowed to the surface rather than the population. Knowing when the sample is ready A protein ready for crystallization trials is one whose identity, mass, oligomeric state and stability are known; which elutes as a single symmetric peak on size exclusion; which shows a narrow size distribution by dynamic light scattering; which is concentrated in a simple buffer; and which has been prepared in enough quantity for several screens and some optimization. For a first screen of a few hundred conditions using nanolitre drops, a hundred microlitres of protein at around 10 milligrams per millilitre is ample. Optimisation will need more. The time invested in this stage is rarely wasted. It is far easier to adjust a construct boundary than to optimize a crystal of a heterogeneous protein, and experienced crystallographers often say that when a project stalls at crystallization, the right move is usually to return to the protein rather than to set up yet another screen. The next chapter describes what happens when a well-prepared protein meets a precipitant in a vapor diffusion drop. Chapter 3: Vapor Diffusion and the Phase Diagram A crystal forms when molecules find it more favourable to be packed against each other in a lattice than to remain dissolved. For that to happen the solution must be supersaturated: it must contain more protein than it can hold at equilibrium. But supersaturation alone is not enough. A solution only slightly supersaturated can sit indefinitely without forming crystals, because the first tiny ordered cluster, the nucleus, costs more in surface energy than it gains in packing. A solution heavily supersaturated nucleates readily but often too readily, producing showers of microcrystals or disordered precipitate. Growing a good crystal is a matter of reaching the region where nucleation happens, and then letting growth proceed slowly in a region where it does not happen again. Vapor diffusion is the most widely used method for doing this, and understanding why it works requires the phase diagram. The protein phase diagram Imagine a graph with protein concentration on one axis and precipitant concentration on the other. The precipitant may be a salt such as ammonium sulfate, a polymer such as polyethylene glycol, or an organic solvent such as 2-methyl-2,4-pentanediol. As precipitant increases, the solubility of the protein falls, and the solubility curve divides the graph into an undersaturated region below it, where crystals would dissolve, and a supersaturated region above it. The supersaturated region is itself divided into zones. Just above the solubility curve lies the metastable zone, where existing crystals grow but new nuclei rarely form within any practical time. Further above lies the nucleation zone, sometimes called the labile zone, where nuclei form spontaneously. Beyond that lies the precipitation zone, where protein comes out of solution as amorphous aggregate faster than it can organise. The boundaries between these zones are not sharp lines; they are regions where the probability of an event changes over experimental timescales. They also depend on temperature, pH, additives and the protein itself, which is why every protein has its own diagram and why the diagram is almost never mapped in full. The ideal crystallization trajectory crosses into the nucleation zone just far enough to form a few nuclei, then retreats into the metastable zone as growth consumes protein and lowers its concentration in solution. Few nuclei and slow growth give few, large, well-ordered crystals. The art of crystallization, in phase-diagram terms, is steering that trajectory. There is a subtlety worth knowing. For many proteins, particularly under polyethylene glycol conditions, there is also a region in which the solution separates into two liquid phases, one rich and one poor in protein. This liquid-liquid phase separation often appears in drops as oily droplets. It is sometimes a dead end, but it is also sometimes the route by which crystals nucleate, because the protein-rich phase is highly concentrated. Pierre Vekilov and others have argued that for some proteins, nucleation proceeds through dense liquid clusters rather than directly from solution. The practical lesson is that phase separation in a screen is not necessarily bad news, and conditions near it are worth exploring. How vapor diffusion moves a drop through the diagram In a vapor diffusion experiment, a small drop containing protein mixed with crystallization solution is sealed in a chamber with a much larger reservoir of the crystallization solution alone. Because the drop starts with a lower concentration of precipitant than the reservoir, usually half as much if equal volumes of protein and reservoir were mixed, water evaporates from the drop and condenses into the reservoir until the vapor pressures equalise. The drop shrinks, and both the protein and the precipitant in it become more concentrated. At equilibrium, for non-volatile precipitants, the drop's precipitant concentration approaches that of the reservoir, and the protein concentration approximately doubles. In phase-diagram terms, the drop travels along a diagonal line, up and to the right, from its starting point towards a final point. If that path crosses into the nucleation zone, crystals may form, and as they grow they deplete protein, pulling the drop's state back towards the solubility curve. This built-in gradual concentration is the great merit of vapor diffusion: the drop sweeps through a range of conditions over hours to days, rather than sitting at one point. The rate of equilibration matters. It depends on the drop volume, the distance between drop and reservoir, the reservoir composition, and the temperature. Drops with high salt reservoirs equilibrate faster than those with polyethylene glycol, because salts lower the vapor pressure of water more strongly per unit concentration. Small drops equilibrate faster than large ones. Slower equilibration generally favours fewer nuclei, and a classic optimization is to slow the process, for example by layering oil over the reservoir or increasing the drop volume. Volatile precipitants break the simple picture. Organic solvents such as ethanol, isopropanol and MPD can themselves diffuse through the vapor phase, so the drop's final composition is not simply the reservoir's. This is not a reason to avoid them, but it means that reproducing a hit exactly requires reproducing the geometry. Following one drop A concrete calculation makes the geometry tangible. Suppose a protein at 10 milligrams per millilitre is mixed in equal volumes with a reservoir containing 1.6 molar ammonium sulfate. At the moment of mixing the drop contains protein at 5 milligrams per millilitre and ammonium sulfate at 0.8 molar. Over the next day or two, water leaves the drop until its salt concentration approaches that of the reservoir. The drop's volume falls to roughly half, and the protein reaches about 10 milligrams per millilitre at 1.6 molar salt, assuming none has yet come out of solution. Now change one variable at a time. If the drop is set with two parts protein to one part reservoir, the starting point moves: protein begins at about 6.7 milligrams per millilitre and salt at about 0.53 molar, and at equilibrium, when the salt has tripled to 1.6 molar, the protein would reach roughly 20 milligrams per millilitre. The drop travels further in protein concentration for the same final precipitant level. If instead the ratio is one part protein to two parts reservoir, the protein starts lower, at about 3.3 milligrams per millilitre, and ends near 5. Varying the drop ratio is therefore a cheap way of sampling different paths through the same diagram without making new solutions, and many optimization protocols set up two or three ratios side by side as a matter of routine. The endpoint of the calculation is only the destination if nothing happens on the way. In a drop that nucleates halfway through equilibration, growing crystals begin consuming protein while the precipitant is still rising. The drop's state then bends away from the straight diagonal, moving towards the solubility curve. A drop that forms a few nuclei early and grows them slowly ends near the curve with a few large crystals; a drop that overshoots deep into the nucleation zone before anything forms ends with a shower of small ones, because nucleation occurs everywhere at once. The same final condition can produce either outcome depending on the speed of the approach, which is why equilibration rate is a genuine experimental variable and not a nuisance. The calculation also explains a common frustration: a hit found at 20 degrees fails to reproduce after the tray is moved to a cold room, or a hit found with one batch of protein fails with the next. Both temperature and small differences in protein concentration shift the starting point or the solubility curve. If a hit sits close to a boundary, small shifts cross it. Recording protein concentration precisely, and measuring it after the final concentration step rather than estimating it, removes one of the commonest sources of irreproducibility. Hanging drops and sitting drops Vapor diffusion is implemented in two main geometries. In the hanging drop format, the drop is placed on the underside of a siliconised glass coverslip, which is inverted over a well containing the reservoir and sealed with grease. The drop hangs from the glass. Hanging drops are easy to observe, easy to access for harvesting by removing the coverslip, and for decades were the default in academic laboratories. Their drawbacks are that they are hard to automate, that drops larger than a few microlitres can fall, and that grease and coverslips make them labour-intensive to set up in large numbers. In the sitting drop format, the drop rests on a shelf or in a small well beside the reservoir, and the plate is sealed with transparent tape. Sitting drops are well suited to robotic dispensing and to standard plate formats with 96 conditions and one to three drop positions per condition. Almost all initial screening in 2026 is done in sitting drop plates set by robots that dispense drops as small as 50 to 200 nanolitres of protein plus an equal volume of reservoir. Drops that small use little protein, allowing several hundred conditions to be tested with a few tens of microlitres of sample. The two formats generally give similar results, but not identical ones. Drop shape, surface contact and equilibration rate differ, and a hit found in a sitting drop does not always reproduce in a hanging drop, or vice versa. When scaling up a nanolitre hit to larger drops for optimization, it is sensible to set up both formats and to expect some variation. Other ways across the diagram Vapor diffusion is not the only method, and some proteins crystallize better by others. Microbatch crystallization, developed and popularised by Naomi Chayen and colleagues in the 1990s, mixes protein and precipitant directly at their final concentrations under a layer of oil, which prevents evaporation. The drop starts and stays at a single point in the phase diagram. Because there is no gradual concentration, conditions must be chosen to fall directly in the nucleation zone, and the method is less forgiving in screening. But it gives excellent control, and a mixture of paraffin and silicone oils allows slow evaporation when desired. Free-interface diffusion places protein and precipitant solutions in contact, often in a capillary or microfluidic channel, and lets them mix by diffusion. A gradient of conditions forms along the interface, sampling a range of the phase diagram in one experiment. Counter-diffusion, championed by Juan Manuel García-Ruiz, extends the idea in long capillaries where the precipitant diffuses into a protein-filled column over days to weeks. Dialysis allows the protein solution's composition to be changed gradually across a membrane, which is useful for proteins that crystallize at low ionic strength. For membrane proteins, crystallization in lipidic cubic phase has become a distinct method in its own right. Ehud Landau and Jürg Rosenbusch introduced it in 1996, using monoolein, which forms a bicontinuous cubic phase with water: a continuous lipid bilayer folded into a three-dimensional periodic surface, interpenetrated by continuous water channels. A membrane protein reconstituted into this bilayer can diffuse within it and, when a precipitant is added, form type I crystals in which molecules stack in layers much as they sit in a membrane. Martin Caffrey's laboratory developed much of the practical methodology, and Vadim Cherezov and colleagues adapted it with robotic dispensing of nanolitre boluses, which made it central to the wave of GPCR structures beginning with the beta-2 adrenergic receptor in 2007. Lipidic cubic phase crystals are often small and hard to see, but they tend to diffract well. Table 1 summarises the main methods and where each tends to be useful. Table 1. Main crystallization methods compared. Method Path through phase diagram Typical scale Strengths Limitations Sitting-drop vapor diffusion Gradual concentration of drop 100 nL to 2 µL drops Robot-friendly, standard for screening Harvesting from wells can be awkward Hanging-drop vapor diffusion Gradual concentration of drop 1 to 10 µL drops Easy viewing and harvest, classic for optimization Manual, hard to automate Microbatch under oil Fixed point, no equilibration 0.1 to 2 µL drops Precise control, simple Needs conditions already in nucleation zone Counter-diffusion Gradient along capillary Capillary columns Samples many conditions, slow growth Slow, specialised hardware Lipidic cubic phase Precipitant added to lipid mesophase 25 to 200 nL boluses Membrane-like environment, well-ordered crystals Viscous handling, small crystals, hard to see Screening: sparse matrices and systematic grids For a new protein there is no way to predict its phase diagram, so the first experiment is a search. The most influential idea in how to search was the sparse matrix screen published by Joseph Jancarik and Sung-Hou Kim in 1991. Rather than systematically vary each chemical parameter, they assembled 50 conditions that sampled combinations of precipitants, buffers, pH values and salts known to have produced crystals of other proteins. The screen was deliberately biased towards what had worked before. Hampton Research commercialised it as Crystal Screen, and it established the model for the dozens of commercial screens that followed. Modern screens fall into a few families. Broad sparse-matrix screens sample the historically successful chemical space. Systematic screens vary precipitant and pH on a grid. Specialised screens target membrane proteins, protein-nucleic acid complexes, or particular classes of precipitant. Some are designed by analysing large databases of successful conditions, such as those compiled in the Biological Macromolecule Crystallization Database. Others incorporate unusual components such as mixtures of small organic molecules intended to mediate crystal contacts. Many laboratories begin with two or three complementary 96-condition screens, totalling several hundred conditions, and examine the pattern of results before investing further. Screening is usually performed at one or two temperatures, most often around 20 degrees Celsius and 4 degrees Celsius. Temperature changes solubility, and a protein that precipitates at room temperature may crystallize in the cold, or the reverse. Temperature is also a variable that can be exploited later in optimization. Reading a screen The results of a screen are not binary. Drops are inspected over days to weeks, increasingly by automated imagers that photograph each drop on a schedule, and each drop is classified as clear, precipitate of various kinds, phase separation, microcrystals, needles, plates, or three-dimensional crystals. The pattern across a screen is informative. If nearly every drop is clear, the protein concentration is probably too low or the protein too soluble under these conditions. If nearly every drop shows heavy brown precipitate, it is too high, or the protein is unstable. A screen in which a quarter to a half of drops show some precipitate is usually well pitched. Light, granular precipitate differs from heavy, brown, amorphous precipitate. The former sometimes indicates microcrystalline material and is worth examining closely or using for seeding; the latter usually indicates denaturation. Skins on the drop surface indicate protein denaturing at the air-water interface. Distinguishing protein crystals from salt crystals is an old problem. Salt crystals form readily in conditions containing phosphate with calcium or magnesium, or high concentrations of sulfate. Several tests help. Protein crystals are soft and crush easily when touched with a fine tool, while salt crystals are hard and click. Protein crystals are usually weakly birefringent or not at all, depending on symmetry, while many salt crystals are strongly birefringent. Protein crystals take up small dyes such as Izit, which stains protein blue, because of their solvent channels. And protein crystals fluoresce under ultraviolet light because of their tryptophan residues, which is why many automated imagers now include UV imaging. More sophisticated methods include second-order nonlinear imaging of chiral crystals, known as SONICC, which detects very small crystals and discriminates chiral protein crystals from achiral salts. Ultimately, the definitive test is diffraction. Many beamlines now allow crystals to be screened in situ, directly in the crystallization plate, without harvesting. A few seconds of X-ray exposure through the plate reveals whether the object diffracts like a protein crystal, with closely spaced spots at low angles, or like a salt, with a few widely spaced strong spots. What a hit means A hit in a screen is a point in chemical space where something ordered formed. It is rarely a condition that yields diffraction-quality crystals as it stands. Initial hits typically produce needles, thin plates, clusters, or small crystals with poorly defined faces. The value of the hit is that it identifies a neighbourhood of the phase diagram worth exploring. The next chapter is about that exploration. Before moving on, it is worth dwelling on one point. The screen is an experiment about the protein, not just a lottery. A protein that gives many hits across chemically different conditions is behaving well and has a broad crystallization neighbourhood; optimizing it will probably be quick. A protein that gives a single marginal hit is on the edge, and it may be more productive to revisit the construct, following the ideas of the previous chapter, than to push that single hit through dozens of rounds of optimization. Many experienced crystallographers adopt the rule that if two or three good screens give nothing useful, the protein must change, not the screen. Hashtags: #ProteinCrystallography #MacromolecularCrystallography #XRayCrystallography #ProteinCrystallization #VaporDiffusion #ProteinPhaseDiagram #CrystalOptimization #Cryoprotection #CrystalHarvesting #SynchrotronRadiation #XRayDiffraction #DiffractionData #StructureFactors #ElectronDensityMaps #PhaseProblem #PhaseDetermination #MolecularReplacement #AnomalousScattering #SADPhasing #MADPhasing #SelenomethioninePhasing #DensityModification #ProteinStructureDetermination #StructuralBiology #FutureOfProteinCrystallography

Download the Book (PDF): Introduction Every day, a large health system runs thousands of uncontrolled experiments. One hospitalist orders normal saline for a patient with sepsis while the physician across the corridor orders lactated Ringer's. One primary care clinic mails stool-test kits to overdue patients; the clinic down the road waits for patients to ask. One cardiologist prescribes 81 milligrams of aspirin after a stent; a colleague prescribes 325. Each of these choices is defensible. Each is made in good faith. And for most of them, nobody knows which option is better, because the variation is haphazard, undocumented as a choice, and never analysed as the natural experiment it is. The pragmatic clinical trial begins from a simple observation about this state of affairs. If clinicians are already choosing between reasonable alternatives, and if nobody knows which alternative is better, then replacing the haphazard choice with a random one costs patients almost nothing and gains society a great deal. The same order sets, the same nurses, the same electronic health record, the same billing and discharge data can be turned into the apparatus of a randomized trial. The research does not sit beside care, in a separate clinic with its own coordinators and case report forms. It runs inside care, using the machinery that care already depends on. That idea is easy to state and hard to execute. The chapters that follow are about the execution: the concrete design decisions that determine whether a trial embedded in routine operations produces a clear answer or an expensive shrug. The subject is not a single technique but a set of interlocking choices, and the argument of this book is that those choices cannot be made one at a time. An embedded trial is a piece of engineering in which the unit of randomization, the consent model, the strategy for getting clinicians to deliver the assigned intervention, and the source of the outcome data all constrain one another. Choose cluster randomization and you have changed what consent can look like. Choose a waiver of consent and you have committed yourself to an intervention that stays within minimal risk. Choose an endpoint drawn from billing records and you have accepted a particular kind of measurement error that shrinks your effect toward zero. The trials that succeed are the ones whose designers saw these connections in advance. Why this matters now Conventional randomized trials remain the best tool for asking whether a new drug works under controlled conditions. They are also slow, costly, and often uninformative about what happens once a treatment leaves the trial clinic. Their participants are selected for adherence and freedom from complicating illness. Their protocols specify visits, tests and dose adjustments that ordinary practice will never replicate. Their outcomes are measured by trained assessors on schedules no clinic keeps. When such a trial shows benefit, a clinician treating an eighty-year-old with kidney disease, heart failure and a complicated medication list is left to guess whether the result applies. Meanwhile, three developments have made a different kind of trial feasible. The first is the near-universal adoption of electronic health records in hospitals and clinics in high-income countries, which means that eligibility, treatment and many outcomes are now recorded in structured, queryable form as a by-product of care. The second is the growth of research networks built on those records, such as the NIH Health Care Systems Research Collaboratory in the United States, launched in 2012, and PCORnet, the national network funded by the Patient-Centered Outcomes Research Institute. The third is regulatory movement. The revised Common Rule, the United States regulation governing federally funded human research, took effect in 2019, and in December 2023 the Food and Drug Administration issued a final rule allowing institutional review boards to waive or alter informed consent for minimal-risk clinical investigations, a power that had previously existed only for research outside the FDA's jurisdiction. In January 2025 the International Council for Harmonisation adopted a revised good clinical practice guideline, E6(R3), which explicitly accommodates pragmatic designs and routine-care data sources. The results have already changed practice. In Sweden, a trial randomizing 7,244 patients with heart attack through a national coronary registry showed that routine thrombus aspiration did not reduce mortality, and it did so at a fraction of the cost of a conventional trial. At Vanderbilt University Medical Center, investigators randomized entire intensive care units, month by month, between balanced crystalloids and saline and enrolled 15,802 critically ill adults without collecting individual written consent. They found a small reduction in a composite of death, new dialysis and persistent kidney dysfunction with balanced fluids, and hospital fluid practice shifted accordingly. In the United States, a PCORnet trial of 15,076 patients with cardiovascular disease compared two aspirin doses with enrolment through patient portals and outcome capture from health records and claims. Not every embedded trial has succeeded, and the failures are at least as instructive as the successes. A trial of longer haemodialysis sessions enrolled 7,035 patients in 266 dialysis units and then had to stop because the intervention facilities barely lengthened their sessions: the average difference between arms was nine minutes. A trial of an online suicide prevention programme delivered through patient portal messaging found that one of its interventions increased self-harm. These were not failures of randomization. They were failures, or surprises, in the parts of the trial that embedding makes hardest to control. What the reader will find here The book is written for researchers who know what a randomized trial is but have not designed one inside a working health system: clinical investigators moving from explanatory trials, health services researchers who have worked mainly with observational data, informaticians asked to support trials, ethicists and IRB members reviewing consent waivers, and health system leaders deciding whether to let their operations become a laboratory. Chapter 1 sets out what pragmatism means as a design attitude, from the 1967 distinction between explanatory and pragmatic trials to the PRECIS-2 tool that lets designers locate a trial on a continuum, and it argues that the central pragmatic choice is the choice of question. Chapter 2 treats the health system itself as trial infrastructure, examining what electronic records, registries and operational partnerships can and cannot provide. Chapter 3 addresses the unit of randomization, explaining when and why whole clinics, wards or hospitals are randomized instead of individual patients, and what that decision costs in statistical efficiency. Chapter 4 goes further into the analysis of cluster and stepped-wedge trials, where errors are common and consequential. Chapter 5 examines consent: the legal criteria for waiving or altering it, the ethical arguments on each side, and the practical alternatives between full written consent and no consent at all. Chapter 6 turns to the clinicians who actually deliver the intervention and the problem of adherence that sank more than one well-designed trial. Chapter 7 deals with outcomes drawn from real-world data, including what measurement error in those data does to a trial's answer. Chapter 8 covers the running of an embedded trial: monitoring, safety oversight, reporting, and what a health system should do with the answer once it has one. The conclusion draws out what follows for health systems that want to learn systematically from their own work. Four tables appear where comparison is genuinely easier in tabular form. The rest is prose, because the subject is a chain of reasoning more than a catalogue. A note on scope The examples come mostly from the United States, the United Kingdom and the Nordic countries, because that is where most published embedded trials have been conducted and where their regulatory frameworks are best documented. The principles travel, but the rules do not always: the conditions for waiving consent differ between jurisdictions, and the European Union's Clinical Trials Regulation, for instance, contains its own provision for simplified consent in certain cluster trials of authorised medicines. Where regulation is discussed, readers working elsewhere should treat the discussion as a map of the questions to ask their own authorities, not as a statement of the answers. The book also takes a position. It holds that most health systems are not currently doing enough randomized learning, that many decisions made on the basis of expert opinion or before-and-after comparisons could be made on the basis of randomized evidence at modest cost, and that the ethical case for embedding research in care is stronger than its critics allow, provided the trials are designed with the discipline this book describes. It also holds that a badly designed pragmatic trial is worse than none, because it produces a confident null result about an intervention that was never actually tested. The difference between those outcomes lies almost entirely in design, and design is the subject of what follows. Chapter 1: From Explanatory to Pragmatic In 1967 two French statisticians, Daniel Schwartz and Joseph Lellouch, published a short paper in the Journal of Chronic Diseases that still frames the field. Their example was a drug intended to make tumours more sensitive to radiotherapy. The drug had to be given for about a month before radiation began. How should a trial compare radiotherapy with and without the drug? One answer is to delay radiotherapy by a month in both arms, giving the drug to one group and a placebo to the other. That design isolates the biological effect of the drug: everything else, including the timing of radiation, is held equal. The other answer is to compare the two strategies as they would actually be used: immediate radiotherapy in one arm, the drug followed by delayed radiotherapy in the other. That design answers the question the clinician faces, which is not "does this molecule sensitize tumours?" but "should I use this approach instead of starting radiation now?" Schwartz and Lellouch called the first attitude explanatory and the second pragmatic. Their point was not that one is right and the other wrong. It was that they answer different questions, that the choice between them should be made deliberately, and that a trial which mixes them carelessly answers neither question well. That point remains the foundation of everything in this book. Two questions, not two qualities A common misreading treats "pragmatic" as a synonym for "cheap", "sloppy" or "observational with a coin toss". The better reading is that an explanatory trial asks can this intervention work under ideal conditions? while a pragmatic trial asks does this intervention work when used as it would be used in ordinary care, compared with what ordinary care would otherwise do? Each question has its own legitimate design. An explanatory trial wants a homogeneous population of patients most likely to respond, because heterogeneity dilutes the signal. It wants high adherence, so it screens out people unlikely to take their pills and monitors them closely. It wants a comparator that differs from the intervention in only one respect, often a placebo. It wants outcomes measured precisely, often physiological markers assessed by blinded investigators. Every one of these choices sharpens the answer to the efficacy question. A pragmatic trial reverses almost every choice, for equally principled reasons. It wants the population that would actually receive the intervention if it were adopted, including the elderly, the multimorbid and the poorly adherent, because that is who the decision affects. It accepts whatever adherence ordinary care produces, because non-adherence is part of what the intervention will face in practice. It compares against usual care, or against another active strategy in current use, because that is the real alternative. And it measures outcomes that matter to patients and health systems, often from data collected as part of care, because those are the outcomes on which adoption decisions turn. Notice that the pragmatic answer is not a degraded version of the explanatory one. If a drug lowers blood pressure by ten points in a trial with 95 percent adherence but patients in ordinary practice take only half their doses because of side effects, the explanatory trial overstates what adopting the drug will achieve. The pragmatic trial tells the health system what it is buying. Conversely, if a pragmatic trial shows no benefit, it cannot say whether the drug fails biologically or fails because nobody takes it. Each design is blind where the other sees. Modern trial methodology gives this old distinction a more precise vocabulary. The ICH E9(R1) addendum on estimands, finalised in 2019, asks trialists to specify exactly what treatment effect they are estimating: in which population, for which variable, how events after randomization such as stopping treatment or switching are handled, and with what summary measure. Pragmatic trials typically target what the addendum calls a treatment policy estimand, in which the effect of being assigned to a strategy is measured regardless of whether patients stay on it. Explanatory trials more often target a hypothetical estimand, which asks what would have happened had everyone adhered. Framing the choice this way forces designers to say out loud what they previously left implicit, and it exposes a common error in embedded trials: collecting data suitable for a treatment policy question and then presenting the results as though they answered an efficacy question. PRECIS-2 and the continuum In practice no trial is purely explanatory or purely pragmatic. A trial may enrol broadly but measure outcomes with trained assessors, or deliver the intervention flexibly but follow patients with research visits. To make these trade-offs visible, an international group led by Kevin Thorpe and Merrick Zwarenstein published the Pragmatic-Explanatory Continuum Indicator Summary, or PRECIS, in 2009. A revised version, PRECIS-2, followed in the BMJ in 2015, led by Kirsty Loudon. PRECIS-2 scores a trial on nine domains, each from 1 (very explanatory) to 5 (very pragmatic). The scores are plotted on a wheel so that a design team can see at a glance where their trial sits and whether its domains pull in consistent directions. The tool is not a quality score. A trial that scores 5 on every domain is not better than one that scores 3; it is answering a different question. The value lies in the conversation the tool forces, particularly between investigators who assume they agree on the purpose of their trial and discover they do not. As Table 1 sets out, each domain corresponds to a specific design decision, and each has a recognisably explanatory and a recognisably pragmatic pole. Table 1. The nine PRECIS-2 domains, with the explanatory and pragmatic extremes of each (adapted from Loudon and colleagues, BMJ 2015). Domain Very explanatory Very pragmatic Eligibility Narrow, likely responders only Everyone who would get it in practice Recruitment Extra effort, advertising, incentives Through usual care contacts only Setting Specialist centres Settings where it would be used Organisation Extra staff, training, resources Existing staff and resources Flexibility of delivery Strict protocol, monitored Delivered as clinicians see fit Flexibility of adherence Close monitoring and correction No measures beyond usual care Follow-up Frequent research visits No more than usual care Primary outcome Surrogate or mechanistic measure Outcome that matters to patients Primary analysis Per protocol or adherent subgroup All data, intention to treat Three domains deserve particular attention in embedded trials, because they are where the embedding itself exerts pressure. The first is organisation. A trial that installs extra nurses, pharmacists or coordinators to deliver its intervention has changed the setting it claims to represent. If the health system will not fund those people after the trial ends, the result does not describe what the system can achieve. This is why many embedded trials deliberately use existing staff, even though it makes the intervention harder to deliver faithfully. Chapter 6 returns to the cost of that choice. The second is flexibility of adherence. Explanatory trials chase adherence; embedded trials often cannot, because the clinicians delivering the intervention are not research staff and the patients are not research volunteers in any active sense. When a trial imposes no adherence measures beyond usual care, it accepts that some of the assigned treatment will not be delivered. That is legitimate if the question is about the effect of a policy. It is disastrous if the policy is so rarely implemented that the arms become indistinguishable, which is exactly what happened in the haemodialysis session trial mentioned in the introduction. The third is follow-up. An embedded trial usually relies on data generated by care: return visits, admissions, prescriptions, deaths. It adds no research visits. This is the source of much of its efficiency, and also the source of a class of measurement problems that Chapter 7 examines in detail. Patients who seek care outside the health system disappear from its records. Outcomes that no clinician records, such as how a patient feels on an ordinary Tuesday, cannot be measured at all without some supplementary mechanism. What makes a question suitable for embedding If pragmatism is a matter of question as much as design, the first task of an embedded trial is to find a question that suits the method. Experience across the Collaboratory, PCORnet and similar networks suggests a recognisable profile. The best questions concern choices that clinicians or health systems are already making, between options already in use, where genuine uncertainty exists about which is better. Fluid choice in intensive care, aspirin dose after coronary disease, the timing of dialysis sessions, the content of a radiology report, the method of reminding patients about cancer screening: all are decisions made thousands of times a day, all involve alternatives that are already considered acceptable, and in each case credible experts disagreed about which alternative was better. That profile has three advantages. It makes the trial ethically straightforward, because patients in either arm receive care that falls within accepted practice. It makes the intervention easy to implement at scale, because the health system already knows how to deliver both options. And it makes the answer immediately actionable, because the health system can switch its default the day the results are known. Questions about new drugs, devices or procedures not yet in routine use fit the embedded model less comfortably. They can sometimes be embedded, as the Salford Lung Study showed in 2016 and 2017 when it tested a newly licensed inhaler combination against usual care across general practices in one English city, using linked electronic records to follow patients. But such trials usually require individual consent, often require extra safety monitoring, and push several PRECIS-2 domains back toward the explanatory end. The more novel the intervention, the more the trial must add to ordinary care to protect patients and measure what it needs. The worst questions for embedding are those where the outcome is not recorded in routine data, where the comparison involves options no clinician would choose between, or where the intervention requires skills or resources the health system does not have. A trial of a psychotherapy that only a few trained specialists can deliver, measured by a symptom scale nobody administers in practice, is not made pragmatic by conducting it in a hospital. A second criterion concerns scale. Embedded trials are efficient per patient but they often need very large numbers, because the effects of choosing between two reasonable options tend to be small. A treatment that reduces a common adverse event by a tenth, from 15 percent to 13.5 percent, would be a meaningful improvement for a health system treating hundreds of thousands of patients a year, yet detecting it requires thousands of patients per arm even with individual randomization, and considerably more if clusters are randomized. The feasibility question for an embedded trial is therefore not only "can we implement this?" but "does the system see enough eligible patients over a reasonable period to answer the question?" Answering it requires counting eligible encounters in historical data before the protocol is finalised, a step that is easy to do in an embedded setting and too often skipped. Equipoise at the level of the system Clinical equipoise, the state of genuine uncertainty within the expert community about the comparative merits of the interventions under study, is the conventional ethical warrant for randomization. In embedded trials it takes a distinctive form, because the uncertainty often concerns a system-level choice as much as an individual one. Consider the Vanderbilt fluid trials, known as SMART in the intensive care units and SALT-ED in the emergency department. Before them, individual clinicians held firm views about balanced crystalloids versus saline, and those views differed. Observational studies suggested harm from saline's chloride load, but a large randomized trial in New Zealand had found no difference in kidney injury. The institution had no policy. The haphazard variation described at the start of this book was the status quo. In that context, randomizing units to one fluid or the other did not deprive any patient of a treatment known to be better. It replaced one form of arbitrariness, the individual preference of whoever happened to write the order, with another form, a coin toss, that could be analysed. This reframing matters for two reasons. It supports the ethical case for embedding, which Chapter 5 examines in detail. And it identifies a practical test for whether a question is ready: if the health system's clinical leaders would be content to adopt either option as the default tomorrow, and genuinely do not know which to choose, the question is suitable. If they already believe one option is better and are asking for a trial only to confirm it, they are likely to undermine the trial through subtle non-adherence in the arm they disfavour. Equipoise at the level of the leaders who control the order sets is as important as equipoise in the literature. A related consideration is what happens after the trial. A pragmatic trial that no one intends to act on is a poor use of patients' participation, however minimal their burden. The strongest embedded trials are designed with a decision in view: the system has agreed in advance that it will adopt whichever option proves superior, or retain the cheaper option if neither proves superior. That agreement shapes the choice of primary outcome, the size of effect worth detecting, and the timetable. It also changes the trial's status in the organisation. The trial is not an academic project tolerated by operations; it is how operations will make a decision it would otherwise make on weaker evidence. The shape of a pragmatic protocol Pulling these threads together, a protocol for an embedded trial differs from a conventional protocol less in its sections than in its emphasis. It specifies the decision the trial will inform and the stakeholders who will make it. It defines eligibility in terms that can be applied from routine data, often by an algorithm run against the electronic record, and it states how many patients that algorithm identified in previous years. It describes the intervention as a change to a default, an order set, a workflow or a message, and explains how the change will be built and switched on. It specifies the unit of randomization and justifies it. It specifies the consent model and the regulatory basis for it. It describes how delivery of the intervention will be measured from routine data, and what the investigators will do if delivery falls short. And it defines outcomes in terms of specific data elements, with a plan for validating them. Every one of those elements involves a trade-off that the following chapters examine. The order is deliberate. The infrastructure of the health system comes first, because it determines what is possible. The unit of randomization comes next, because it shapes both the statistics and the ethics. Consent follows, then adherence, then outcomes, then conduct. At each step the question is the same one Schwartz and Lellouch asked: what decision is this trial meant to inform, and does each design choice serve that decision? Chapter 2: The Health System as Trial Infrastructure A conventional trial builds its own infrastructure. It recruits sites, hires coordinators, writes case report forms, trains assessors, ships drug supplies and installs a database. When the trial ends, most of that apparatus is dismantled. An embedded trial borrows its infrastructure from the health system: the registration desk that identifies patients, the electronic record that documents care, the pharmacy that dispenses, the order sets that shape prescribing, the billing system that records procedures and admissions, the registries that capture outcomes. The appeal is obvious. The difficulty is that this infrastructure was built for other purposes, by people with other priorities, and it behaves accordingly. This chapter looks at what the machinery of routine care can do for a trial, where it falls short, and what kind of partnership between researchers and operational leaders is needed to use it well. Registries: the proof of concept The clearest early demonstration that routine data could carry a randomized trial came not from electronic health records but from a national clinical registry. The Swedish Coronary Angiography and Angioplasty Registry, known as SCAAR, records every coronary angiogram and angioplasty performed in the country, with data entered by operators during or immediately after the procedure. By the late 2000s it had near-complete coverage and was linked, through Sweden's personal identity numbers, to national registers of deaths and hospital admissions. The Thrombus Aspiration in ST-Elevation Myocardial Infarction in Scandinavia trial, TASTE, used that infrastructure directly. Patients having angioplasty for a major heart attack were identified in the registry's online interface; the operator obtained brief oral consent and clicked to randomize the patient to manual thrombus aspiration followed by stenting, or to stenting alone. The registry form, which operators filled in anyway, captured the baseline data. The national registers captured the primary outcome, death from any cause at 30 days, with no loss to follow-up. Between 2010 and 2013 the trial enrolled 7,244 patients. Death at 30 days occurred in 2.8 percent of the aspiration group and 3.0 percent of the stenting-only group, a difference well within chance. The result, published in the New England Journal of Medicine in 2013 and later reinforced by the larger TOTAL trial, led guideline bodies to stop recommending routine aspiration. TASTE shows what makes registry-based randomization work. The registry already had high-quality data on the relevant population, collected at the point of care by clinicians who valued it. The outcome was hard, unambiguous and captured in national data. The intervention was a procedural choice that operators were already making in both directions. And the randomization step was built into a form clinicians already used, so the marginal burden of the trial was a few clicks. The same Swedish infrastructure later supported other trials, including DETO2X, which randomized 6,629 patients with suspected heart attack to supplemental oxygen or ambient air and found no mortality benefit from oxygen. Few health systems have registries of this quality, and the ones that do usually have them for a narrow range of conditions: cardiac procedures, joint replacements, cancer, dialysis, intensive care. For most questions the relevant infrastructure is the electronic health record. What the electronic record can do An electronic health record can perform four functions for a trial, each with its own strengths and failure modes. The first is identification of eligible patients. Eligibility rules that depend on structured data, such as age, diagnosis codes, laboratory values, prescriptions and location of care, can be encoded as an algorithm, sometimes called a computable phenotype, and run automatically. The algorithm can fire in real time, prompting a clinician when a patient meets criteria, or in batches, producing lists for outreach. The Suicide Prevention Outreach Trial used exactly this approach: patients who answered the ninth item of the Patient Health Questionnaire, the item about thoughts of death or self-harm, with a response indicating frequent suicidal thoughts during routine depression screening were identified from the records of four integrated health systems, and 18,882 were randomized between 2015 and 2018. No research staff screened patients; the record did it. The weakness of computable eligibility is that it inherits every quirk of documentation. Diagnosis codes reflect billing incentives as well as clinical reality. A problem list may carry a diagnosis resolved years ago. Laboratory values from outside laboratories may sit in scanned documents the algorithm cannot read. An algorithm validated at one institution may perform differently at another because clinicians there document differently. Before a trial relies on a computable phenotype, someone must check a sample of flagged and unflagged records by hand to estimate how often the algorithm is wrong and in which direction. The second function is randomization and delivery. Many modern record systems allow an assignment to be generated inside the record and linked to the patient or encounter, so that the patient's arm can drive what the clinician sees: a different default in an order set, a different best-practice alert, a different template for a report. In the Vanderbilt fluid trials, the order entry system presented the assigned fluid as the default for each unit according to the month's allocation. Clinicians could override it, but the default shaped behaviour. This is powerful, but it depends on the health system's informatics team, who have competing priorities, strict change control and a justified fear of anything that might disrupt clinical workflows. The third function is documentation of what happened. The record captures orders, administrations, dispensings, procedures and encounters, which lets investigators measure delivery of the intervention without asking anyone. In the haemodialysis session duration trial, TiME, session length was drawn from the dialysis provider's operational database, which is how the investigators discovered that their intervention was not being delivered at the intended intensity. The fourth function is ascertainment of outcomes. Admissions, deaths recorded in hospital, laboratory results, new diagnoses and prescriptions are all captured as a by-product of care. The limits of that capture are the subject of Chapter 7, but one limit belongs here because it shapes the choice of infrastructure from the start. A single health system's record sees only the care delivered within that system. A patient enrolled in a trial at an academic medical centre who is admitted with a stroke to a community hospital across town will not appear in the centre's record unless someone looks elsewhere. Integrated delivery systems that both insure and treat their members, such as Kaiser Permanente or the Veterans Health Administration, see much more of their patients' care than open systems do, which is one reason so many early embedded trials were conducted in them. Networks and common data models A single health system can host a trial of a common problem, but many questions need more patients or more diversity than any one system provides. Multi-system trials bring a new difficulty: each system's record is organised differently, uses different local codes, and is managed by a different team. The standard response is a common data model, a shared specification into which each participating system translates its data. PCORnet, established by the Patient-Centered Outcomes Research Institute beginning in 2013 and 2014, maintains such a model, and its participating networks refresh their translated data regularly and run standard data quality checks. The FDA's Sentinel system and the OMOP model maintained by the OHDSI collaborative serve similar functions for surveillance and observational research. For a trial, a common data model means that an eligibility query or outcome definition can be written once and run across all sites, and that results come back in a consistent format. The ADAPTABLE trial, which compared 81 milligrams and 325 milligrams of daily aspirin in patients with established cardiovascular disease, was PCORnet's flagship demonstration. Patients were identified from the records of participating health systems using a computable phenotype, invited by letter, email or portal message, and enrolled and randomized online, with electronic consent. They reported their own hospitalizations through a web portal or by telephone, and these reports were supplemented and checked against the common data model and, for Medicare beneficiaries, against claims data. The trial enrolled 15,076 patients and followed them for a median of 26.2 months. It found no significant difference in the composite of death, hospitalization for myocardial infarction or hospitalization for stroke (7.28 percent with 81 milligrams against 7.51 percent with 325 milligrams), and no significant difference in major bleeding. ADAPTABLE is instructive precisely because it was a hybrid. Its eligibility, recruitment and outcome ascertainment were embedded in routine data. But its intervention was not a change in how clinicians practised; it was an instruction to patients about which tablet to buy. Adherence depended on patients, and patients assigned to 325 milligrams switched doses far more often than those assigned to 81 milligrams, 41.6 percent against 7.1 percent. That asymmetry complicates interpretation, and it illustrates a general truth: embedding one part of a trial does not embed the others. Each function the health system performs for a trial must be considered separately. Point-of-care trials inside the record Before the large networks existed, the Veterans Health Administration tested a more radical idea: that a trial could be run entirely through the clinician's ordering workflow. In the VA's point-of-care trial of insulin regimens, reported by Louis Fiore and colleagues in 2011, a physician ordering insulin for an eligible hospitalized patient was presented with a prompt noting that the choice between sliding-scale and weight-based insulin was uncertain and asking whether the patient could be approached for the trial. Consented patients were randomized within the record, and the assigned regimen populated the order. Outcomes were drawn from the record. The trial's main lesson was about recruitment. Enrolment proceeded slowly, largely because clinicians did not always respond to the prompt and because obtaining consent still required a separate conversation. The technical integration worked; the human integration was harder. That lesson recurs through this book. The informatics of embedding is rarely the binding constraint. The binding constraint is almost always the interaction between the trial's demands and the time, attention and priorities of the people delivering care. In the United Kingdom, the RECOVERY trial of treatments for hospitalized patients with COVID-19 showed what happens when that constraint is relaxed by necessity and design together. Launched in March 2020, RECOVERY used a short consent form, a single-page electronic randomization form completed by clinicians, and follow-up drawn largely from national hospital and death records through NHS data services, with a brief form completed at 28 days. Its protocol was deliberately minimal: it collected only what was needed to answer the question. Within weeks it had recruited thousands of patients across nearly every acute hospital in the country. In June 2020 it reported that dexamethasone reduced mortality among patients receiving respiratory support, and in the same programme it showed that hydroxychloroquine and lopinavir-ritonavir did not help. RECOVERY was not fully pragmatic in every PRECIS-2 domain; patients gave consent, and the interventions were drugs not yet in routine use for the disease. But it demonstrated that a health system's routine machinery, combined with a radically simplified protocol, could deliver a very large randomized trial at speed. The operational partnership Every example above depended on a relationship that conventional trial methodology barely mentions: the relationship between investigators and the people who run the health system. Operational leaders control the resources an embedded trial needs. The chief medical information officer decides whether the informatics team will build the randomization logic and when. The nursing director decides whether a new bathing protocol will be taught to ward staff. The pharmacy director decides whether a formulary change can be made for half the wards. The compliance office decides how patient data may be used. None of these people is primarily accountable for research, and each has reasons to be cautious. A change to an order set that confuses clinicians can harm patients. A research build that breaks during an electronic record upgrade can consume weeks of an informatics team's time. The trials that succeed treat these leaders as partners in the question, not as providers of services. The ABATE Infection trial, which compared routine bathing with chlorhexidine bathing plus nasal decolonisation in non-intensive care units at 53 hospitals, was conducted with HCA Healthcare, a large hospital company whose infection prevention leadership co-designed the trial and whose corporate structure allowed a standard protocol to be rolled out across many hospitals at once. The trial's predecessor, REDUCE MRSA, had shown in 2013 that universal decolonisation in intensive care units reduced resistant organisms and bloodstream infections, and HCA had an operational interest in knowing whether to extend the practice to general wards. The question was the company's question as much as the investigators'. That alignment made the trial possible. A useful test of the partnership is to ask what would happen to the trial if its principal investigator left. If the answer is that it would collapse, the trial is a research project hosted by the health system. If the answer is that operations would continue running it because they want the answer, it is genuinely embedded. Leora Horwitz and colleagues at NYU Langone Health described in 2019 one version of this partnership taken to its conclusion. The health system created a unit to run rapid, randomized tests of its own operational changes, such as the wording of letters and the design of outreach calls, using routine data for outcomes and completing each test in months. In their first ten trials, a minority of the changes tested improved the target outcome. The finding that most well-intentioned operational changes did not work as hoped is itself a strong argument for testing them, and the speed of the programme showed that randomized learning need not be slow or expensive when the system itself owns the questions. Governance and the fragility of borrowed infrastructure Borrowed infrastructure can be taken back. Health systems upgrade their electronic records, merge with other systems, change billing vendors, reorganise clinics and revise order sets for reasons unrelated to any trial. Each change can disrupt an embedded trial in ways a conventional trial rarely experiences. A redesigned order set can silently remove the default that delivered the intervention. A new laboratory information system can change the codes that define an outcome. A clinic closure can remove a cluster mid-trial. Three practices reduce this fragility. The first is to register the trial's dependencies with the informatics team as formally as any clinical system, so that a planned change triggers a check of whether the trial is affected. The second is to monitor the trial's data streams continuously, looking for sudden changes in eligibility counts, intervention rates or outcome rates that signal a broken pipe rather than a real effect. The third is governance: a steering group that includes operational leaders with the authority to protect the trial's infrastructure when competing priorities arise. None of these practices is expensive. All of them are routinely neglected, because conventional trial training does not mention them. The health system, in short, is a powerful but temperamental host. It can supply patients, data and delivery at a scale no conventional trial can match, but only if the trial is designed around the system's actual behaviour. The first design decision shaped by that behaviour is the unit of randomization, which is the subject of the next chapter. Chapter 3: Choosing the Unit of Randomization In a conventional trial the unit of randomization is almost always the individual patient. Each person who consents is assigned by chance to one arm or another, and statistical theory for analysing such trials is mature and familiar. In embedded trials, individual randomization is often impractical, sometimes impossible and occasionally misleading. The alternative is to randomize groups of patients, such as everyone treated in a clinic, admitted to a ward, dialysed in a facility or cared for by a particular physician. These groups are called clusters, and trials that randomize them are cluster randomized trials. Cluster randomization is the design decision with the most far-reaching consequences in pragmatic research. It changes the statistical efficiency of the trial, sometimes drastically. It changes who counts as a research participant and therefore what consent means. It changes the risks of bias and the methods needed to analyse the data. This chapter explains why embedded trials so often randomize clusters, what that costs, and which variants of the cluster design suit which problems. The next chapter goes into the analysis. Why randomize groups There are four common reasons to randomize clusters instead of individuals, and it is worth distinguishing them because they carry different implications. The first is that the intervention operates at the level of the group. A change to a unit's bathing protocol, a clinic's staffing model, a hospital's antibiotic stewardship programme or a radiology department's reporting template affects everyone the unit serves. It cannot sensibly be delivered to some patients and withheld from others in the same place. The ABATE Infection trial randomized hospitals because the intervention was a ward-wide bathing routine. The Lumbar Imaging with Reporting of Epidemiology trial, known as LIRE, randomized primary care clinics because its intervention was a short passage of text inserted into spine imaging reports, explaining how common findings such as disc degeneration are in people without back pain. The reports were generated by radiology systems serving whole clinics, and the text either appeared in a clinic's reports or it did not. The second reason is contamination. Even when an intervention could in principle be delivered to individuals, clinicians who treat patients in both arms may carry lessons from one to the other. A physician prompted to deprescribe a sedative for one patient may start deprescribing for all her patients. If contamination is substantial, individual randomization underestimates the intervention's effect, sometimes to the point of missing it. Randomizing physicians or practices keeps each clinician in one arm. The third reason is practicality of delivery. Many embedded interventions work by changing a default, an order set or a standard workflow. Changing the default for a whole unit is simple; making it depend on each patient's randomized assignment requires more elaborate informatics and asks clinicians to keep track of which patients are in which arm. In the Vanderbilt fluid trials, it was far easier for an intensive care unit to stock and default to one fluid for a month than to switch fluids patient by patient. The fourth reason, more contested, is consent. When the intervention is delivered at the level of the cluster and individual patients cannot practically opt out of it, individual consent to randomization may be impossible. As Chapter 5 explains, this is one of the recognised grounds for waiving consent under the relevant regulations. But investigators should be careful not to reverse the logic. Choosing cluster randomization primarily to avoid seeking consent, when individual randomization with consent would be feasible, is an ethically weak justification and will be treated as such by careful review boards. Each reason should be stated explicitly in the protocol, because each has a different implication for the design. If contamination is the concern, clusters must be large enough to contain the clinicians who might contaminate. If the intervention is inherently group-level, the cluster is defined by whatever structure delivers it. If practicality is the concern, a cluster crossover design may recover much of the efficiency lost. The price: correlation within clusters Patients in the same cluster tend to resemble each other more than patients in different clusters. They share physicians, nurses, protocols, local populations and sometimes infections. Their outcomes are therefore correlated. This correlation is measured by the intracluster correlation coefficient, usually abbreviated ICC, which can be understood as the proportion of the total variation in the outcome that lies between clusters rather than within them. An ICC of zero means cluster membership tells you nothing about a patient's outcome; an ICC of one means every patient in a cluster has the same outcome. Correlation within clusters reduces the information each additional patient contributes. If the ICC is high, adding more patients to an existing cluster adds little, because they largely repeat what the cluster's first patients already showed. The standard way to quantify this loss is the design effect, sometimes called the variance inflation factor. For clusters of equal size m and an intracluster correlation ρ, the design effect is 1 + (m − 1)ρ. The sample size needed for an individually randomized trial must be multiplied by this factor to achieve the same power with cluster randomization. The formula is simple and its implications are counter-intuitive. Because the design effect depends on cluster size as well as the ICC, even a small correlation can be expensive when clusters are large. Suppose a trial randomizes 20 clinics, each contributing 100 patients, for 2,000 patients in all, and suppose the ICC is 0.05. The design effect is 1 + 99 × 0.05, which is 5.95. The trial has the statistical information of an individually randomized trial of about 336 patients. If the same clinics each contributed 1,000 patients, the design effect would be almost 51, and the trial of 20,000 patients would be equivalent to an individually randomized trial of about 400. Adding patients to clusters barely helps. Adding clusters helps a great deal. This is the single most important quantitative fact about cluster trials, and it has a practical corollary. In embedded trials the number of available clusters is often fixed by the structure of the health system: a system has 12 hospitals or 40 clinics, and that is that. The number of patients per cluster, by contrast, can be enormous, because routine data capture everyone. Designers who reason from the total number of patients, which looks impressive, rather than from the number of clusters, which is the real constraint, will overestimate their power. In LIRE, with roughly a quarter of a million patients, the effective information depended on the number of clinics and the correlation within them, not on the headline count. How large are ICCs in practice? They vary widely by outcome and setting. For many patient-level outcomes in primary care and hospital settings they are small, frequently below 0.05, and published compilations of ICCs from earlier trials are the best source for planning. Process measures, such as whether a clinician ordered a test or prescribed a drug, typically have much higher ICCs than clinical outcomes, because they reflect local habit directly. That matters in embedded trials whose outcomes are often processes of care. The ICC also depends on how the outcome is measured and on whether cluster-level covariates are adjusted for. Planning with a single borrowed ICC is risky; planning with a plausible range, and showing how power changes across it, is better practice. Unequal cluster sizes, which are the norm in health systems, make matters slightly worse. When some clinics are large and others small, the design effect increases, roughly in proportion to the variation in cluster size. Formulas incorporating the coefficient of variation of cluster size are available and should be used, because a trial planned as though all clusters were equal will be underpowered. Variations on the cluster design The simplest cluster trial is the parallel design: clusters are randomized once to intervention or control and stay there for the duration of the trial. It is easy to explain, easy to implement and robust. Its disadvantage is inefficiency when the number of clusters is small, because comparisons are made only between clusters, and between-cluster differences in patients and practices add noise. Several alternatives exploit the fact that embedded trials can often observe the same clusters over time. As Table 2 sets out, each gains something and gives something up. Table 2. Common cluster randomized designs used in embedded trials and their main trade-offs. Design How allocation works Main advantage Main risk Parallel cluster Each cluster randomized once to one arm Simple, robust to time trends Needs many clusters Cluster crossover Clusters switch arms in randomized periods Within-cluster comparison, high efficiency Carry-over; patients spanning periods Stepped wedge All start as control; switch to intervention in random order Every cluster eventually gets intervention Confounding by time; complex analysis Parallel with baseline Parallel design plus pre-randomization period Adjusts for baseline cluster differences Assumes stable secular trends Covariate-constrained Randomization restricted to balanced allocations Better balance with few clusters Requires matching analysis In a cluster crossover trial, each cluster receives both conditions in different periods, in a randomized order. The Vanderbilt fluid trials are the best-known example. In SMART, five intensive care units at one academic centre were each assigned, month by month, to use either balanced crystalloids or saline, with the sequence determined in advance and units switching repeatedly over nearly two years. Because each unit served as its own control, the stable differences between units, such as their patient mix and staffing, largely cancelled out. With only five units, a parallel design would have been hopelessly underpowered; the multiple crossover made the trial feasible. SMART enrolled 15,802 adults and found that 14.3 percent of patients receiving balanced crystalloids experienced a major adverse kidney event within 30 days, compared with 15.4 percent receiving saline. Crossover designs have their own hazards. If the effect of an intervention persists after it is withdrawn, the control period that follows is contaminated. That is rarely a concern for intravenous fluids, whose effects on a new cohort of patients do not depend on what the unit did last month, but it would be a serious concern for an educational intervention or a change in clinician habits. Patients whose stay spans a switch complicate analysis. And the design requires clusters that can switch conditions cleanly and repeatedly, which fits a ward's fluid stock better than a clinic's care management programme. In a stepped-wedge design, all clusters begin in the control condition, and at regular intervals a randomly chosen group of clusters crosses to the intervention until, by the end, all clusters receive it. The design, formalised by Michael Hussey and James Hughes in 2007, has attractions that explain its popularity in embedded research. It is easy to sell to health system leaders who are reluctant to withhold a promising intervention from half their sites, because every site eventually gets it. It suits interventions that the system cannot roll out everywhere at once for practical reasons, since the rollout has to be staggered anyway. And it allows comparisons both between and within clusters. LIRE used a stepped wedge: its clinics moved from standard reports to reports containing the epidemiological text at randomly determined times across the trial period. The stepped wedge's central weakness is that intervention status is confounded with calendar time. By design, the intervention is more common late in the trial than early. If outcomes are changing for other reasons, as they almost always are in health systems, a naive analysis attributes the secular trend to the intervention. The analysis must model time explicitly, and the validity of the result depends on getting that model right. Stepped wedges also take longer than parallel designs, are sensitive to delays in implementation at individual sites, and can be less efficient than a parallel design when the ICC is small. The consensus view among methodologists is that the stepped wedge is valuable when a staggered rollout is unavoidable or politically necessary, but it should not be chosen merely because it sounds more acceptable. When only a small number of clusters is available, simple randomization can produce arms that differ markedly by chance in size, patient mix or baseline outcome rates. Covariate-constrained randomization addresses this by listing all possible allocations, discarding those that are badly imbalanced on important cluster characteristics, and choosing randomly among the remainder. Stratification or matching on key characteristics serves a similar purpose. These methods improve balance, but the analysis must account for them, typically by adjusting for the constraining covariates, and the constraint must not be so tight that only a handful of allocations remain, which would compromise the randomization itself. Identification and recruitment bias Cluster randomization introduces a form of bias that individually randomized trials largely avoid. In an individually randomized trial, patients are identified and consent before their allocation is known, so the allocation cannot influence who enters the trial. In many cluster trials, clusters are randomized first and patients are identified afterwards, by clinicians who know which arm their cluster is in. If the intervention changes who is identified as eligible, the arms can differ in composition for reasons unrelated to chance. Consider a trial randomizing clinics to an enhanced depression care programme, where eligible patients are those whom clinicians diagnose with depression. Clinicians in intervention clinics, newly attentive to depression, may diagnose milder cases; clinicians in control clinics continue as before. The intervention arm then contains a less severely ill population, and its outcomes look better regardless of whether the programme works. This problem, variously called identification bias or recruitment bias, has been documented in published cluster trials and can be large. Embedded trials have a distinctive defence against it: eligibility can often be defined from data recorded independently of the intervention, before or at the moment of randomization. If eligibility depends on a diagnosis made in a previous year, on a laboratory value, on a procedure code or on admission to a unit, the intervention cannot influence who enters. The strongest embedded cluster trials exploit this. They define their population by an algorithm applied to routine data, run identically in both arms, using information that the intervention cannot touch. SMART's population was every adult admitted to a participating intensive care unit. The suicide prevention trial used screening questionnaire responses recorded as part of routine visits. Where eligibility must depend on post-randomization clinical judgement, investigators should compare the characteristics and numbers of identified patients across arms, and treat any imbalance as a warning. Choosing the cluster Defining the cluster is itself a design decision. Candidates in a health system include individual clinicians, care teams, clinics, wards, hospitals, dialysis facilities, pharmacies and geographic areas. Smaller clusters give more clusters and less design effect but more risk of contamination, because patients and clinicians move between them. Larger clusters contain contamination but are fewer and more variable. The right cluster is usually the smallest unit that fully contains the intervention and its likely spillovers. If the intervention is a change in a ward's nursing protocol and nurses rarely float between wards, the ward is the natural cluster. If nurses float extensively, the hospital may be necessary. If the intervention is a change to a clinician's order set and patients see only one clinician, the clinician may suffice; if patients see several, the practice is safer. Mapping these flows from routine data before finalising the design, using scheduling records and encounter data to see how often clinicians and patients cross between candidate clusters, is one of the simplest and most useful pieces of preparatory work an embedded trial can do. One further consideration is the stability of the cluster over the trial. Clinics merge, wards are repurposed, physicians leave. A trial whose clusters dissolve partway through must decide how to handle the patients affected, and that decision is easier to make in the protocol than in the middle of the trial. The unit of randomization shapes almost everything that follows. It determines the statistical methods the analysis will need, which is the subject of the next chapter. It determines who is a research participant and what consent is possible, which is the subject of Chapter 5. And it determines where the levers of adherence lie, which is the subject of Chapter 6. A decision taken early for practical reasons will return, again and again, as a constraint. Hashtags: #PragmaticClinicalTrials #EmbeddedResearch #LearningHealthSystems #RoutineHealthcare #ElectronicHealthRecords #RealWorldData #ClusterRandomizedTrials #ClusterCrossover #SteppedWedgeTrials #PRECIS2 #TreatmentPolicyEstimand #PointOfCareTrials #RegistryBasedTrials #PCORnet #ComputablePhenotypes #RoutineCareData #ConsentWaivers #MinimalRiskResearch #InterventionAdherence #PragmaticOutcomes #IntraclusterCorrelation #IntentionToTreat #HealthSystemPartnerships #TrialInfrastructure #FutureOfPragmaticTrials

Download the Book (PDF): Introduction A kinesin molecule walking along a microtubule takes steps of 8 nanometres and stalls when it is pulled back with a force of roughly 6 to 7 piconewtons. A strand of double-stranded DNA held at both ends behaves like a floppy entropic spring below about 10 piconewtons, stiffens sharply as it approaches its full length, and then, near 65 piconewtons, suddenly lengthens by about 70 per cent in a cooperative transition that is still debated in its molecular details. A viral packaging motor can drive DNA into a capsid against a resisting force of more than 50 piconewtons. Every one of these numbers was obtained by holding a microscopic plastic or glass bead in a focused beam of infrared light and watching how far the bead moved. That is the whole trick of the optical trap, and it is deceptively simple. A tightly focused laser beam pulls a dielectric particle toward the brightest point of the focus. Displace the particle a little and it experiences a restoring force proportional to the displacement, so the trap acts as a spring. If the spring constant is known and the displacement can be measured, the force follows from Hooke's law. With stiffnesses around 0.01 to 1 piconewton per nanometre and position resolution that, in the best instruments, reaches a few ångströms, the method spans exactly the range of forces and distances over which proteins and nucleic acids do their mechanical work. The difficulty is that each link in that chain, from the laser to the reported force, can quietly go wrong. A beam that is slightly misaligned in the back aperture of the objective produces a trap that is weaker in one direction than another. A detector calibrated on one bead and used on another of slightly different diameter reports displacements that are off by ten per cent or more. A trap calibrated at the centre of a flow cell and used two bead radii from the coverslip has a drag coefficient nearly 40 per cent larger than the value in free solution. A tether that appears to be a single DNA molecule can be two, and a force that appears to be the stall force of a motor can be the rupture force of a weak antibody linkage. None of these errors announces itself. The data look clean and the plots look convincing. The controlling idea This booklet rests on a single claim: an optical trap is a force transducer whose numbers are only as trustworthy as the chain of alignment, surface chemistry, and calibration standing behind them, and that chain must be built and checked deliberately at every link. The laser does not measure force. The trap does not measure force. What measures force is an instrument whose optical path, detector response, bead geometry, hydrodynamic environment, and molecular attachment have each been characterised well enough that their errors can be stated. The piconewton is earned, not read off a screen. This view has practical consequences. It means that alignment is not a one-time installation chore but a measurable property of the instrument that drifts and must be verified. It means that bead coating is not a kitchen recipe but a determinant of which forces the experiment can reach before the construct fails at the wrong place. It means that calibration is an experiment in its own right, with its own controls, statistical uncertainty, and systematic errors, and that a calibration done carelessly can dominate the uncertainty in every downstream result. And it means that a published force should come with enough information about how it was obtained that another laboratory could reproduce it, a standard the field has only recently begun to discuss openly. Who this is for and how it is organised The reader I have in mind is a graduate student, postdoctoral researcher, or scientist moving into single-molecule work from biochemistry, structural biology, or physics, who needs to understand what the instrument is doing well enough to run it responsibly, and a technically curious non-specialist who wants to know how such small forces can be measured at all. The mathematics is kept to what is needed: Hooke's law, the Stokes drag on a sphere, the thermal energy kBT, and the Lorentzian power spectrum of a bead jiggling in a harmonic well. Numbers are worked out in the text where they matter. The chapters follow the order in which an experiment is actually built. Chapter 1 explains the physics of trapping: why a focused beam pulls on a bead, why the result behaves as a spring, and what sets the stiffness. Chapter 2 walks through the design and alignment of a working instrument, from laser choice to beam steering to the conjugate planes that let a trap move without changing its strength. Chapter 3 addresses detection: how back-focal-plane interferometry converts bead displacement into a voltage, and how the noise floor of the whole instrument is characterised and lowered. Chapter 4 turns to the beads themselves, polystyrene and silica, and the surface chemistry used to coat them and to passivate the flow cell. Chapter 5 covers the molecular tethers and handles that connect a biological molecule to the beads, and how to verify that a tether is what it appears to be. Chapter 6 is the heart of the matter: calibrating trap stiffness by equipartition, power spectral analysis, drag, and direct momentum measurement, with the corrections each method needs. Chapter 7 describes how forces and motions are actually measured, in force clamps, position clamps, and constant-velocity pulling, and what landmark experiments on DNA, motors, and folding have shown. Chapter 8 takes up analysis, error budgets, and reproducibility. The Conclusion draws out what follows for practice. A note on scope The subject here is the classical single-beam and dual-beam optical trap used with micron-sized beads, which is how the large majority of single-molecule force spectroscopy is done. Related techniques appear where they illuminate a point: magnetic tweezers, which apply constant force to many molecules at once and excel at torque; atomic force microscopy, which reaches higher forces with stiffer probes; and acoustic force spectroscopy, which applies forces with standing ultrasonic waves. Optical torque wrenches, holographic multi-trap systems, and trapping inside living cells are mentioned but not treated in depth. Protocols are given with concrete parameters where those are standard in the field, but any protocol should be adapted to local reagents, instruments, and safety rules. Class 4 infrared lasers can blind a person before the injury is noticed, because the beam is invisible and the eye has no blink reflex to it. Nothing in this booklet replaces laser safety training, interlocks, and appropriate eyewear rated for the wavelength in use. The historical frame is brief but worth stating. Arthur Ashkin at Bell Laboratories showed in 1970 that radiation pressure from a laser could accelerate and hold micron-sized particles, and in 1986, with Joseph Dziedzic, John Bjorkholm, and Steven Chu, demonstrated the single-beam gradient trap that is the basis of modern optical tweezers. In 1987 he trapped viruses and live bacteria. By the early 1990s the laboratories of Steven Block, James Spudich, Carlos Bustamante, and others had turned the technique into a quantitative instrument for molecular mechanics, and in 2018 Ashkin shared the Nobel Prize in Physics for the invention. The method is now also available commercially in integrated instruments that combine dual traps, microfluidics, and confocal fluorescence. That commercial availability makes the argument of this booklet more pressing, not less: when the instrument arrives calibrated from the factory, the temptation to treat its numbers as given is strongest. Chapter 1. How Light Holds a Bead Light carries momentum. A beam of power P travelling through a medium of refractive index n carries momentum at a rate of nP/c, where c is the speed of light in vacuum. The number is small in everyday terms but not in molecular ones. A 100 milliwatt beam in water, with n about 1.33, delivers momentum at a rate of 0.1 × 1.33 / (3.0 × 10⁸) newtons, or about 440 piconewtons. If a transparent bead redirects even a few per cent of that momentum flux, the reaction force on the bead is several piconewtons, which is the scale at which a single motor protein works. The whole of optical trapping follows from this bookkeeping: whenever a particle bends or scatters light, the change in the light's momentum appears as an equal and opposite force on the particle. It is conventional to describe that force as the sum of two parts. The scattering force pushes the particle along the direction the beam is travelling, because light that is reflected or scattered backward or sideways has lost forward momentum. The gradient force pulls the particle toward the region of highest intensity, provided the particle has a higher refractive index than its surroundings. A stable three-dimensional trap exists only where the gradient force along the beam axis is strong enough to overcome the scattering force that tries to push the particle downstream. That condition is what made Ashkin's 1986 single-beam trap a genuine invention: it required focusing the light so steeply that the axial gradient could win. Two pictures of the same force For a particle much larger than the wavelength of light, geometric ray optics gives an intuitive account. Imagine a glass bead a few micrometres across sitting slightly to the left of the focus of a converging beam. Rays passing through the bead are refracted twice, once on entering and once on leaving. Because the beam is brighter on its axis than at its edges, more light passes through the right side of the bead than the left, and the refraction of that stronger bundle bends light toward the left. Light deflected leftward has gained leftward momentum, so the bead feels a push to the right, back toward the axis. The same argument along the axis explains why steeply converging rays pull a bead back toward the focus: the refracted rays emerge more nearly parallel to the axis than they entered, gaining forward momentum, and the bead recoils backward toward the focus. The steep rays at the edge of a high numerical aperture cone contribute most of this axial restoring force, which is why trapping objectives have numerical apertures of 1.2 or higher. For a particle much smaller than the wavelength, the Rayleigh regime, the bead behaves as an induced point dipole. The electric field of the light polarises it, and a polarised particle in a spatially varying field is drawn toward the region of strongest field, just as a scrap of paper is drawn toward a charged comb. The gradient force in this regime scales with the cube of the particle radius and with the factor (m² − 1)/(m² + 2), where m is the ratio of the particle's refractive index to that of the medium. The scattering force scales with the sixth power of the radius, so for very small particles the gradient term dominates easily; as particles grow, the scattering term grows faster and trapping becomes harder along the axis. The beads used in biophysics are inconveniently in between. A polystyrene bead of 0.5 to 2 micrometres diameter is comparable in size to the wavelength of a 1064 nanometre laser in water, which is about 800 nanometres. Neither the ray picture nor the dipole picture is quantitatively correct here, and exact calculation requires generalised Lorenz–Mie theory or numerical electromagnetic methods, often combined with a model of the aberrated focus actually produced by the objective. Alexander Rohrbach showed in 2005 that such calculations could reproduce measured trap stiffnesses closely when the real focus was modelled carefully, but in routine practice nobody predicts stiffness from first principles. The theory is used to understand trends, and the stiffness is measured. Why the trap behaves as a spring The central practical fact is that near its equilibrium position a trapped bead experiences a restoring force proportional to its displacement. Along one lateral axis, F = −κx, where κ is the trap stiffness. This is not a law of optics but a consequence of expanding any smooth potential well to second order around its minimum. For micron-sized beads in a well-aligned trap, the linear region extends over roughly one to two hundred nanometres from the centre, which comfortably covers the displacements used in most molecular measurements. Beyond that, the force continues to rise but more slowly, reaches a maximum somewhere around a bead radius from the centre, and then falls off as the bead escapes. The maximum force the trap can exert is the escape force, which is typically determined experimentally by dragging a bead through fluid at increasing speed until it falls out. Two properties of the stiffness matter constantly. First, κ is proportional to laser power at the specimen, because every force in the problem is proportional to intensity. Doubling the power doubles the stiffness and doubles the maximum force. Second, the stiffness is different along different axes. The two lateral stiffnesses are usually within ten or twenty per cent of each other if the beam is circular and well aligned, with the residual difference coming from the polarisation of the light, which elongates the focal spot slightly along the polarisation direction. The axial stiffness is typically several times weaker than the lateral stiffness, because the focus is longer along the axis than across it, and because the scattering force partly opposes the axial gradient. The bead's equilibrium position also sits slightly downstream of the geometric focus for the same reason. It is useful to fix typical numbers early. A 1 micrometre polystyrene bead held with a few tens to a few hundreds of milliwatts at the specimen will have lateral stiffnesses in the range of roughly 0.05 to 0.5 piconewtons per nanometre. With a stiffness of 0.1 piconewtons per nanometre and a linear range of 150 nanometres, the trap can apply forces up to about 15 piconewtons while remaining a simple spring. To reach 50 or 100 piconewtons, which is necessary for overstretching DNA or for resisting strong motors, one must use more power, larger beads, or both, and accept that the linear range and the heating of the solvent become harder to manage. The trapping efficiency, often written Q and defined through F = QnP/c, captures the fraction of the momentum flux converted into restoring force. For polystyrene beads in water, lateral values of Q in the range of a few hundredths to around a tenth are typical of real instruments. The refractive index contrast sets much of this. Polystyrene has a refractive index near 1.57 and silica near 1.45, against 1.33 for water. The higher contrast of polystyrene gives stronger trapping for a given size and power, which is one reason polystyrene beads became the workhorse of the field. Silica beads trap more weakly but are denser, easier to functionalise with some chemistries, and less prone to certain kinds of nonspecific adsorption. Particles with a refractive index lower than the medium, such as air bubbles, are pushed out of the focus rather than drawn in. Metallic particles scatter and absorb strongly and can be trapped only under particular conditions, usually as small nanoparticles, and they heat up considerably. Wavelength, absorption, and the cost of light Almost all biological trapping uses near-infrared light, between about 800 and 1100 nanometres, because this window combines low absorption by proteins, nucleic acids, and water with the availability of stable, powerful lasers. Within that window the choice of wavelength is a trade-off among three quantities: heating of the aqueous medium, photodamage to biological material, and the practical performance of lasers and detectors. Table 1 summarises the common options. Table 1. Common trapping wavelengths and their practical trade-offs (after Neuman and Block 2004; Neuman et al. 1999; Peterman et al. 2003). Wavelength Typical source Water heating Photodamage in cells Practical note 830 nm Diode laser Lowest of the three Local minimum Limited power; beam quality varies 970–980 nm Diode or fibre laser Highest of the three Local minimum Heating limits usable power 1064 nm Nd:YVO4, Nd:YAG, or fibre laser Moderate, about 8 K per W in water Moderate Watts available; silicon detectors lose sensitivity The heating deserves a concrete number. Erwin Peterman, Frederick Gittes, and Christoph Schmidt measured in 2003 that, with 1064 nanometre light, the temperature at the focus in water rose by about 8 kelvin per watt of laser power, and they showed that the heat comes mainly from absorption by the water around the bead rather than by the bead itself. A trap run at 500 milliwatts at the specimen therefore raises the local temperature by about 4 kelvin. That is not destructive, but it is not negligible: water viscosity falls by roughly 2 per cent per kelvin near room temperature, so a 4 kelvin rise lowers the viscosity by around 8 per cent, which enters directly into any calibration that uses the drag coefficient. Enzymes are also temperature-sensitive; a rate measured in the trap is a rate at the trap's temperature, not at the nominal room temperature written in the methods section. Photodamage is a separate matter. Keir Neuman and colleagues in Steven Block's laboratory used a tunable laser and rotating tethered Escherichia coli cells as a sensitive assay in 1999 and found that the action spectrum for damage had minima near 830 and 970 nanometres and maxima near 870 and 930 nanometres. The damage disappeared under anaerobic conditions, pointing to oxygen-dependent photochemistry, and its dependence on intensity was linear, suggesting a single-photon process. For in vitro work with purified molecules the analogous concern is the generation of reactive oxygen species that can damage proteins and nucleic acids in or near the focus. Later work by Markita Landry and colleagues in 2009 showed that tethered DNA in optical traps can be damaged by photoactivated singlet oxygen, particularly when fluorescent dyes are present, and that oxygen scavenging systems substantially extend tether lifetimes. The practical lesson is that oxygen scavengers are not only for fluorescence experiments; they belong in many pure force experiments as well. A worked design exercise Suppose an experiment needs to hold a DNA tether at forces up to 20 piconewtons while resolving a protein's steps of a few nanometres, using 1 micrometre polystyrene beads and a 1064 nanometre laser. The first question is what stiffness to aim for. The linear range of the trap sets an upper limit on displacement of perhaps 150 nanometres, and the detector's linear range, discussed in Chapter 3, is usually tighter still, often around 100 nanometres for this bead size. To keep 20 piconewtons within 100 nanometres of displacement, the stiffness must be at least 0.2 piconewtons per nanometre. The second question is what stiffness costs in resolution and bandwidth. A stiffer trap confines Brownian motion more tightly, reducing the root-mean-square fluctuation to √(kBT/κ), which at 0.2 piconewtons per nanometre is √(4.1/0.2), or about 4.5 nanometres. It also responds faster: the bead relaxes toward equilibrium with a characteristic frequency κ/(2πγ), where γ is the drag coefficient. For a 1 micrometre bead in water at 25 °C, the Stokes drag is γ = 6πηr, which with η about 0.89 millipascal seconds and r = 0.5 micrometres gives about 8.4 × 10⁻⁹ newton seconds per metre, or 8.4 × 10⁻⁶ piconewton seconds per nanometre. The corner frequency at 0.2 piconewtons per nanometre is therefore 0.2 / (2π × 8.4 × 10⁻⁶), close to 3.8 kilohertz. Events faster than a fraction of a millisecond will be blurred by the bead's own response, whatever the detector bandwidth. The third question is how much light the stiffness requires. That depends on the instrument's efficiency, which is not known until it is measured, but a well-built trap of this kind commonly needs on the order of one or two hundred milliwatts at the specimen to reach 0.2 piconewtons per nanometre for a micron bead. Because objectives transmit only part of the infrared light that enters them, sometimes well under 70 per cent at 1064 nanometres for objectives designed for the visible, and because steering optics and filters lose more, the laser output may need to be several times larger. The heating at 200 milliwatts at the specimen is about 1.6 kelvin, which is tolerable but should be accounted for. The final design question is whether the experiment would be better served by a dual-trap geometry, where the DNA is suspended between two trapped beads far from any surface, rather than tethered between a trapped bead and the coverslip. For step detection at the few-nanometre level the answer is usually yes, for reasons of drift and surface interaction examined in the next chapters. The exercise illustrates a pattern that recurs throughout the booklet: every choice about force range is simultaneously a choice about noise, bandwidth, heating, and the geometry of the assay, and none can be made in isolation. From physics to instrument requirements The physics sets a short list of requirements that the next chapter translates into hardware. The beam must be focused by a high numerical aperture objective, and the back aperture of that objective must be filled with light appropriately so that the steep marginal rays exist; too narrow a beam gives a weak axial trap, while greatly overfilling wastes power that is clipped by the aperture. Mahdieh Mahamdeh, Citlali Pérez Campos, and Erik Schäffer showed in 2011 that for a given laser power the optimum is often a moderate underfilling rather than the heavy overfilling that older guidance recommended, a reminder that rules of thumb in this field are worth testing on one's own instrument. The focus must be as free of aberration as possible at the depth where the experiment is done. With an oil-immersion objective focusing into water, the refractive index mismatch introduces spherical aberration that grows with depth, so the trap weakens and elongates as it moves further from the coverslip. A trap calibrated at one depth is not calibrated at another. Water-immersion objectives largely avoid this problem at the cost of a lower maximum numerical aperture, and some oil objectives include correction collars or are used with immersion oils of adjusted index for a particular working depth. The laser must be stable in power and pointing, because any fluctuation in either appears as a fluctuation in stiffness or in trap position, which is indistinguishable from a molecular event. The beam must be steerable, so that the trap can be moved to apply force or to follow a motor, without the trap's strength changing as it moves. And the instrument must provide a way to measure bead position with nanometre or better precision at bandwidths of tens of kilohertz. Each of these requirements has a standard engineering answer, and each answer has characteristic ways of failing. One final point of physics shapes everything that follows. A trapped bead in water is never still. It is continuously buffeted by thermal collisions with solvent molecules, each of which delivers a tiny impulse, and the sum of those impulses drives a random walk that the trap confines. The mean thermal energy per degree of freedom is kBT/2, and at 25 °C, kBT is about 4.1 piconewton nanometres. That quantity is the natural energy unit of single-molecule mechanics: the work done by one piconewton over four nanometres is roughly one kBT. It is also the currency in which the trap's stiffness can be measured, since a bead in a harmonic well of stiffness κ fluctuates with a mean square displacement of kBT/κ along each axis. For κ = 0.05 piconewtons per nanometre, that is 4.1/0.05, or about 82 square nanometres, a root-mean-square excursion of about 9 nanometres. The Brownian motion that sets the noise floor of every measurement is also the most reliable ruler with which the instrument can calibrate itself, a duality that Chapter 6 exploits in full. Chapter 2. Building and Aligning the Instrument An optical trap is built from ordinary optical components arranged with unusual care. A laser, a few lenses, a steering element, a dichroic mirror, a microscope objective, a condenser, and a position detector are enough to make one. Kits such as the Thorlabs modular optical tweezers system are used in teaching laboratories to do exactly that. What separates a teaching trap from a research instrument capable of resolving the 3.4 ångström steps of RNA polymerase is not exotic hardware but control of the details: the quality of the beam, the placement of every lens relative to the planes it must image, the mechanical and thermal stability of the whole assembly, and a discipline of alignment that is checked quantitatively rather than by eye. This chapter describes the optical layout of a typical single-beam and dual-beam instrument, explains the principle of conjugate planes that governs where each element must sit, and gives an alignment procedure with the checks that confirm it has succeeded. The optical train The laser comes first. For 1064 nanometre work the usual sources are diode-pumped Nd:YVO4 or Nd:YAG lasers and ytterbium fibre lasers, with output powers from one to ten watts or more. What matters is not raw power but a clean single transverse mode, often specified as an M² value close to 1, together with low intensity noise and good pointing stability. Many research instruments couple the laser into a polarisation-maintaining single-mode optical fibre before it reaches the trapping optics. The fibre acts as a spatial filter, delivering a near-perfect Gaussian mode, and converts pointing fluctuations of the laser into power fluctuations at the fibre output, which are easier to monitor and correct. The cost is coupling loss, typically a quarter to a half of the power, and the need to guard against back-reflections into the laser with an optical isolator. Power is controlled with a half-wave plate followed by a polarising beam splitter, which divides the beam into two orthogonally polarised parts whose ratio depends on the wave plate angle. In a dual-trap instrument those two parts become the two traps. Where power must be modulated quickly, for instance to implement feedback, an acousto-optic modulator is used. A small fraction of the beam is often picked off to a photodiode so that power fluctuations can be logged and, if necessary, divided out of the detector signal later. A beam expander, a pair of lenses forming a telescope, then enlarges the beam to the diameter needed to fill the back aperture of the objective appropriately. The objective, usually a 60× or 100× oil-immersion lens with numerical aperture 1.3 to 1.49, or a 60× water-immersion lens with numerical aperture about 1.2, is reached via a dichroic mirror that reflects the infrared and transmits visible light for imaging. The transmission of the objective at the trapping wavelength should be measured, not assumed; objectives optimised for fluorescence can transmit well under three quarters of the incident infrared, and two objectives with the same catalogue number can differ. Above the sample sits the condenser, which in an ordinary microscope provides illumination but in a trap has a second job: it collects the laser light that has passed through the specimen, including the portion scattered by the bead. A second dichroic mirror above the condenser diverts that infrared light to a relay lens that images the back focal plane of the condenser onto a quadrant photodiode or position-sensitive detector. That arrangement, back-focal-plane detection, is the subject of the next chapter. For now the key point is that the condenser's numerical aperture should be high enough to collect most of the scattered light, ideally matching the objective, and that its position along the axis must be adjusted reproducibly. Conjugate planes and beam steering Nearly every alignment problem in an optical trap comes down to a single principle. The position of the trap in the specimen is set by the angle at which the beam enters the back aperture of the objective, not by where it enters. Tilting the beam at the back aperture moves the focus sideways in the specimen; translating the beam across the back aperture tilts the cone of light at the focus but does not move it much, and, crucially, clips part of the beam on the aperture edge, weakening the trap asymmetrically. A beam steering element must therefore sit in a plane optically conjugate to the back focal plane of the objective. When a mirror in such a plane is tilted, a telescope relays that tilt to the back focal plane as a pure change of angle, with the beam pivoting about the centre of the aperture. The focus moves laterally in the specimen while the beam stays centred on the aperture and the stiffness stays constant. If the steering mirror is placed anywhere else, tilting it also walks the beam across the aperture, and the trap strengthens and weakens as it moves. That error is easy to create and, without a test, invisible. The standard layout uses one or two telescopes of the 4f type, each made of two lenses separated by the sum of their focal lengths, to relay the steering plane to the back focal plane while also setting the beam diameter. Axial position of the trap is adjusted by moving one lens of a telescope along the beam, changing the beam's divergence at the objective and thus the depth of the focus. Several steering technologies are in common use, with the trade-offs summarised in Table 2. Table 2. Beam steering technologies used in optical traps (after Neuman and Block 2004; Moffitt et al. 2008). Element Update speed Range Stiffness uniformity Main caveat Piezo tip-tilt mirror Sub-millisecond Tens of µm Good if conjugate Hysteresis unless closed loop Acousto-optic deflector Microseconds Several µm to tens of µm Varies with drive frequency Diffraction efficiency must be mapped Galvanometer mirror Sub-millisecond Large Good if conjugate Two separate planes for x and y Spatial light modulator Milliseconds Many traps at once Depends on hologram Slow, lossy, pixel crosstalk Acousto-optic deflectors deserve a note because they are widespread in high-resolution instruments. They deflect the beam by diffraction from an ultrasonic wave in a crystal, so the deflection angle is set electronically, with no moving parts and no hysteresis. Their diffraction efficiency, however, varies with the acoustic frequency, so a trap steered across the field by a deflector changes power, and therefore stiffness, as it moves. Careful instruments map that variation and compensate by adjusting the radio-frequency amplitude. Deflectors can also switch between positions fast enough to time-share one beam between several traps, a technique used in multi-trap instruments. Dual-trap instruments deserve particular attention because they have become the standard geometry for high-resolution work. In the most common design the laser is split by polarisation into two beams; one passes via a fixed mirror or a slow steering mirror and the other via a fast steering element; both are recombined at a second polarising beam splitter before the objective. The two traps are thus orthogonally polarised, and after the condenser the light can be separated by polarisation again so that each bead is detected on its own detector. A DNA molecule stretched between the two beads is held far from any surface. Jeffrey Moffitt and colleagues in Carlos Bustamante's laboratory showed in 2006 that recording the difference between the two beads' signals cancels much of the noise that the two traps share, because pointing and drift that move both traps together do not change the distance between them. The same group reported base-pair-scale resolution of transcription and translocation with such instruments. Alignment and verification An alignment procedure What follows is the sequence most laboratories use, with details that vary by design. It assumes that laser safety measures are in place, that alignment is done at the lowest practical power with the beam attenuated, and that infrared viewer cards or a camera sensitive at the trapping wavelength are available. Define the optical axis of the microscope first. Mount alignment irises at the objective port and at a point well upstream, centred on the mechanical axis of the objective turret. All subsequent beams will be aligned to this axis. Align the laser beam, or the fibre output collimator, through the two irises using two steering mirrors. Adjust the upstream mirror to centre the beam on the upstream iris and the downstream mirror to centre it on the downstream iris, and repeat until both are centred without further adjustment. Insert the telescope lenses one at a time, beginning with the lens closest to the objective. After each insertion the beam must still pass through the downstream iris; translate the lens laterally until it does. A lens that is off-centre steers the beam. Set lens spacings to form true 4f telescopes, checking collimation of the output with a shear plate interferometer or by measuring beam diameter at two widely separated points. Place the steering element at the plane conjugate to the objective's back focal plane. A practical test is to tilt the steering element while watching the beam at the back aperture on an infrared card or camera; if the beam spot walks laterally, the element is not at the conjugate plane. With the objective in place and a sample of beads, bring a bead into the trap and adjust the axial telescope lens so that the trap forms at a sensible depth, commonly several micrometres into the sample. Align the condenser by Köhler illumination for imaging, then adjust its height so that the laser light emerging above it is collimated. Centre the image of the condenser back focal plane on the position detector. Verify the alignment quantitatively, as described below, before any experiment. A few failures are common enough to name. Back-reflections from lens surfaces can form faint secondary traps or interference fringes on the detector; tilting optics slightly and using anti-reflection coatings for the trapping wavelength suppresses them. Mirror mounts relax over hours after adjustment, so an instrument aligned in the afternoon may need a check the next morning. And a telescope assembled with lenses in the wrong orientation, curved face the wrong way, adds spherical aberration that reduces stiffness without any visible sign in the beam. Verifying the alignment Alignment should be judged by the trap, not by the beam. Four checks are standard. First, the lateral stiffnesses along x and y, measured by the power spectrum method of Chapter 6, should agree to within roughly 10 to 20 per cent; a larger difference suggests astigmatism, clipping, or an elliptical beam. Second, when the trap is moved across its working range by the steering element, the stiffness should stay constant to within a few per cent; systematic variation indicates that the steering element is not in the conjugate plane or, for an acousto-optic deflector, that its efficiency has not been compensated. Third, when a trapped bead is moved axially through the focus, its image should stay centred; lateral drift with focus means the beam is entering the objective at an angle to its axis. Fourth, the detector signal for a bead held at the trap centre should be close to zero on both axes; a large offset means the condenser back focal plane is not centred on the detector. A fifth check is less often performed but just as informative: record the maximum escape force by moving the stage at increasing speed while a bead is held, and compare it with the value from previous weeks. A drop signals contamination of the objective, a degraded laser mode, or misalignment long before it shows up as an obvious problem in experimental data. Beyond the core optics Adding fluorescence and flow Many current experiments combine trapping with fluorescence imaging, so that force and extension can be recorded at the same time as the binding, movement, or conformational state of labelled proteins. The combination raises two problems that shape instrument design. The first is spectral: the infrared trapping light, the visible excitation light, and the fluorescence emission must be separated by dichroic mirrors and filters with enough rejection that the trapping laser, which is millions of times brighter than a single fluorophore's emission, does not leak onto the camera or photon counters. The second is photophysical. Matthew Comstock, Taekjip Ha, and Yann Chemla reported in 2011 that simultaneous exposure to the intense trapping light and visible excitation accelerates photobleaching of common dyes, apparently through absorption from excited states. Their solution was to interlace the two: the trap and the excitation laser were switched on and off alternately at tens of kilohertz, faster than the bead could respond, so that the bead felt a time-averaged trap while the fluorophore was never exposed to both beams at once. Other groups keep the fluorescence imaging region away from the trap foci, for example by imaging the middle of a long DNA molecule held between two beads with confocal scanning or wide-field illumination. Iddo Heller and colleagues combined dual traps with stimulated emission depletion microscopy in 2013 to resolve densely bound proteins on stretched DNA, showing how far the integration can be pushed. The integrated commercial instruments now common in single-molecule laboratories, such as the LUMICKS C-Trap, pair dual traps with confocal or wide-field fluorescence and a multichannel laminar-flow cell. In that flow cell several streams of buffer run side by side without mixing, because flow at these scales is laminar. A pair of beads is trapped in one stream, moved into a second stream containing DNA to catch a tether, checked in a third stream of buffer alone, and then moved into a fourth containing the protein of interest. Andrea Candelli, Gijs Wuite, and Erwin Peterman described the approach in a 2011 review. The ability to change the chemical environment of a single tethered molecule in seconds is one of the most useful practical advances of the past fifteen years, because it separates tether formation from the experiment and allows the same molecule to be tested in several conditions. Commercial integration changes the alignment task without removing it. The factory aligns and calibrates the instrument, and software reports stiffness and force automatically. The user's responsibilities become checking that the reported calibration is consistent with independent tests, knowing what assumptions the software makes about bead size, viscosity, and distance from surfaces, and recognising when a service visit is needed. Instruments from LUMICKS, from Bruker's JPK line, and from Impetux, whose SENSOCELL module measures force by direct momentum detection, embody different design choices, and understanding which quantities each measures directly and which it infers is part of using them well. Mechanical and environmental stability The last part of building the instrument concerns things that are not optics at all. The whole system must sit on a vibration-isolated optical table, and in the most demanding instruments inside a room with tight temperature control, often to within a small fraction of a kelvin. Steel and aluminium expand at roughly 10 to 25 parts per million per kelvin, so a 10 centimetre mechanical path that warms by 0.1 kelvin lengthens by 100 to 250 nanometres, enough to swamp a molecular signal. Acoustic noise and air currents also move optics and change the refractive index of air along the beam path. Enclosing the optical path is standard. Elio Abbondanzieri and colleagues, in reaching base-pair resolution of RNA polymerase stepping in 2005, went further and enclosed the beam path in helium, whose refractive index varies much less with temperature and pressure fluctuations than air's. A fair summary is that the instrument's performance is set by the weakest element among laser noise, optical alignment, mechanical drift, and detection, and that all four degrade with time. A well-run laboratory keeps a log of calibration and alignment checks, so that when data start to look strange there is a record of when the instrument last behaved as expected. Chapter 3. Detection and the Noise Floor A trap applies force; a detector measures position. Every number the instrument reports, whether a force, an extension, or a step size, comes ultimately from a voltage on a photodetector that has been converted into nanometres. The conversion factor, the linear range over which it holds, the bandwidth of the detector, and the noise sources that corrupt the signal together define what the instrument can see. This chapter explains how position is detected, how the detector is calibrated, and how the noise floor of the instrument is measured and pushed down. It ends with a principle that sounds obvious and is often neglected: the resolution of an experiment is not a property of the instrument alone but of the instrument, the tether, and the averaging time taken together. Three ways to see a bead The simplest detector is a camera. The bead is imaged through the microscope, and its position in each frame is found by fitting its intensity profile or computing a centroid. Sub-pixel precision of a few nanometres is achievable with bright, high-contrast images, and a camera can track many beads at once. Its limitation is bandwidth. Conventional cameras run at tens to hundreds of frames per second, and while modern high-speed CMOS cameras reach many kilohertz over a small region of interest, their precision falls as exposure time shortens because fewer photons are collected. Camera tracking also blurs motion within each exposure, which biases the apparent variance of a moving bead downward, an effect analysed by Wesley Wong and Kenneth Halvorsen in 2006 that matters whenever camera data are used for calibration. The second approach images the bead directly onto a quadrant photodiode or a position-sensitive detector at an image plane. This gives high bandwidth but measures the bead's position in the field of view, so that when the trap moves, the detector must be moved or recalibrated. The third approach, which dominates high-resolution work, is back-focal-plane interferometry. Light from the trapping laser that passes the bead without scattering interferes with light the bead scatters forward. When the bead sits exactly in the centre of the trap, the resulting intensity pattern in the back focal plane of the condenser is symmetric. When the bead moves sideways, the pattern shifts, and a quadrant photodiode placed in a plane conjugate to the condenser's back focal plane registers the asymmetry. Frederick Gittes and Christoph Schmidt gave the theoretical treatment in 1998, showing that the lateral signal is, to first order, proportional to the displacement of the bead relative to the laser focus and independent of where that focus sits in the field of view. That independence is the method's decisive advantage: when the trap is steered, the detector continues to report displacement from the trap centre, which is exactly the quantity needed to compute force. Changes in the total light reaching the detector also carry information about axial position, a principle exploited by Ernst-Ludwig Florin, Arnd Pralle, and colleagues in the photonic force microscope in the late 1990s. The quadrant detector's outputs are combined into normalised differences. If the four quadrants give voltages A, B, C, and D, the lateral signals are formed as (A + B − C − D) and (A + D − B − C), each divided by the sum of all four. Normalising by the sum removes sensitivity to fluctuations in laser power, which is why a power fluctuation does not masquerade as a displacement in a well-designed system, at least to first order. Many instruments use a separate, weak detection laser at a different wavelength, overlapped with the trapping beam, rather than detecting with the trapping light itself. This decouples detection from trapping: the trapping beam can be steered or modulated without changing the detector's response. It also allows the detection wavelength to be chosen for the detector, for example 830 or 850 nanometres, where silicon photodiodes are fast and efficient. Axial detection deserves a brief word of its own. The total intensity reaching the detector changes as the bead moves along the beam, because the interference between scattered and unscattered light depends on the bead's axial position relative to the focus. The resulting signal is less linear and more sensitive to bead size and condenser position than the lateral signals, and its calibration usually requires moving the bead axially with a piezoelectric stage or objective positioner. Most molecular experiments are therefore designed so that the force acts laterally, in the plane of the coverslip, where both trapping and detection are best characterised. The silicon filtering problem That last point hides a trap of its own. Silicon photodiodes absorb 1064 nanometre light only weakly, because the photon energy is close to silicon's band gap. Light penetrates deep into the device, and charge carriers generated outside the depletion region diffuse slowly to the junction. The result is that the detector acts as a low-pass filter with a characteristic frequency that can be as low as a few kilohertz and depends on the light intensity and the bias voltage. Kirstine Berg-Sørensen and colleagues identified and modelled this parasitic filtering in 2003. Its consequence is that power spectra recorded with such detectors fall off at high frequency more steeply than a pure Lorentzian would, and a calibration that ignores the filter overestimates or underestimates the corner frequency depending on how the fit is weighted. There are three remedies. The filter can be measured and included explicitly in the fitting model, which is what careful calibration software does. The silicon detector can be replaced with one designed for the near infrared, such as an indium gallium arsenide quadrant detector or a silicon device with a thick, fully depleted absorption region operated at high reverse bias. Or a separate detection laser at a shorter wavelength can be used, as described above. Whichever route is taken, the detector's frequency response should be checked rather than taken from the datasheet. Converting volts to nanometres The detector sensitivity, commonly written β in nanometres per volt or its inverse in volts per nanometre, converts normalised signal into displacement. It depends on bead size and refractive index, on the trapping or detection wavelength, on the axial position of the bead relative to the focus, on the height and numerical aperture of the condenser, and on the depth of the trap in the sample. It must therefore be determined under the conditions of the experiment, and ideally for each bead, because commercial bead preparations have a diameter spread of several per cent and sensitivity varies steeply with size. Three methods are used. In the first, a bead is stuck to the coverslip and moved through the detection beam in steps with a calibrated piezoelectric stage, while the detector signal is recorded. The resulting curve is linear near the centre and then turns over; its slope at the centre gives the sensitivity and its shape gives the linear range, which for a 1 micrometre polystyrene bead is typically of order ±100 nanometres. The drawback is that a bead stuck to the surface is not the bead used in the experiment, and it sits in a different optical environment. In the second method, a trapped bead is moved rapidly relative to the detection beam, for example by scanning a separate detection laser across it, which allows calibration of the actual bead in place. In the third, the sensitivity is extracted from the power spectrum of the trapped bead's Brownian motion, using the fact that the diffusion constant of a sphere of known size in a fluid of known viscosity is predicted by the Einstein relation. This last method, discussed in Chapter 6, has the great merit of calibrating the actual bead at the actual location, at the cost of relying on knowing the bead radius, the viscosity, and the temperature. Nonlinearity beyond the central region must be handled explicitly. A common practice is to fit a cubic or higher polynomial to the detector response over the full range used, and to discard or flag data in which the bead strays outside the calibrated region. Crosstalk between axes, where motion along x produces some signal on y, arises from imperfect alignment of the detector or the beam and should be measured and corrected with a rotation of the coordinate frame. Sampling, filtering, and aliasing Between the detector and the computer sit amplifiers, filters, and an analogue-to-digital converter, and each shapes the data. The amplifiers must have bandwidth well above the fastest signal of interest, and their gain must be set so that the full range of expected bead motion fills a good part of the converter's range without clipping. A 16-bit converter spanning ±10 volts resolves about 0.3 millivolts; with a typical sensitivity of a few hundred nanometres per volt of normalised signal, that is well below a nanometre, but a gain set too low can easily waste most of that resolution. The sampling rate determines the highest frequency that can be represented, the Nyquist frequency, equal to half the sampling rate. Any signal or noise above the Nyquist frequency that reaches the converter is not simply lost; it is folded back into lower frequencies, a phenomenon called aliasing. Brownian motion has power at all frequencies, falling as 1/f² above the corner frequency, so without an anti-aliasing filter the recorded power spectrum is inflated near the Nyquist frequency. For a trap with a corner frequency of a few kilohertz, sampling at 50 or 100 kilohertz with an analogue low-pass filter set somewhat below the Nyquist frequency keeps aliasing small. Where it cannot be avoided, it can be modelled: Berg-Sørensen and Henrik Flyvbjerg gave in 2004 an exact expression for the aliased Lorentzian that a finite-rate sampler produces, and good calibration software fits that expression rather than the ideal one. Digital filtering and decimation after acquisition are common and legitimate, provided they are documented. A record sampled at 100 kilohertz and averaged down to 1 kilohertz for step analysis is a different data set from the raw record, and the averaging window should be reported. Timestamps matter too. When force data are combined with fluorescence images or with a moving stage, the clocks of the separate devices must be synchronised, usually by recording a shared trigger or by running everything from one acquisition card, or events will be misattributed by milliseconds. A final practical matter is that the raw, high-bandwidth data should be kept. Many analyses that seem unnecessary at the time of the experiment, such as recalibrating from the Brownian motion of a tethered bead or checking for a spectral peak that appeared on a particular day, require the full-rate record. Storage is cheap; repeating a week of experiments is not. Where the noise comes from The noise in an optical trap measurement has a floor that cannot be removed and a set of excess sources that can. The floor is the Brownian motion of the bead itself. A bead of drag coefficient γ experiences a random thermal force whose power spectral density, counting positive frequencies only, is 4kBTγ. Over a measurement bandwidth B well below the trap's corner frequency, the resulting force noise is √(4kBTγB) and the corresponding position noise is that force divided by the stiffness. The numbers are instructive. For a 1 micrometre bead in water with γ = 8.4 × 10⁻⁶ piconewton seconds per nanometre, and at room temperature, the force noise over a 100 hertz bandwidth is √(4 × 4.1 × 8.4 × 10⁻⁶ × 100), or about 0.12 piconewtons. In a trap of stiffness 0.1 piconewtons per nanometre, that is about 1.2 nanometres of position noise. Averaging down to a 10 hertz bandwidth reduces it by √10, to about 0.37 nanometres. Detecting ångström-scale steps is therefore possible, but only if the steps last long enough to be averaged and only if nothing else in the instrument drifts by an ångström during that time. The formula also shows which knobs matter. A smaller bead reduces γ and so reduces the thermal force noise, which is why high-resolution experiments often use beads of 0.5 to 1 micrometre rather than larger ones. A stiffer trap reduces position noise at a given force noise. And in the dual-trap geometry, where the measured quantity is the distance between two beads connected by a compliant tether, the effective stiffness of the system includes the tether, which usually softens the connection and couples the two beads' fluctuations. The noise on a tethered measurement is always worse than the noise on a free bead in the same trap, a point sometimes forgotten when instruments are characterised with untethered beads. The excess noise sources are summarised in Table 3. Each dominates in a different frequency range, and identifying which one limits a given measurement is the first step in improving it. Table 3. Principal excess noise sources in optical trap measurements and standard remedies (after Neuman and Block 2004; Moffitt et al. 2006; Carter et al. 2007). Source Typical time scale Symptom Standard remedy Laser pointing fluctuation Milliseconds to seconds Apparent bead motion correlated with trap Fibre coupling, enclosure, differential detection Laser power fluctuation Broad Stiffness and signal fluctuate Normalised detection, power monitoring Air currents and index fluctuation Milliseconds to seconds Low-frequency wander Enclosed or helium-filled beam path Mechanical and thermal drift Seconds to hours Slow baseline creep Temperature control, dual traps, active stabilisation Acoustic and table vibration Tens to hundreds of Hz Spectral peaks Isolation, quiet room, damped mounts Electronic pickup Mains frequency and harmonics Sharp peaks at 50 or 60 Hz Grounding, shielding, anti-alias filtering Drift requires particular comment because it limits the longest and the most precise experiments. In a surface-coupled geometry, where one end of the molecule is attached to the coverslip, any motion of the stage relative to the trap changes the tether's extension directly. Stage drift of a nanometre per minute is common without active control. Ashley Carter, Thomas Perkins, and colleagues described in 2007 a method of tracking fiducial marks on the coverslip with separate lasers and correcting the stage position by feedback, stabilising the sample to about 0.1 nanometre in three dimensions. The dual-trap geometry avoids surface drift altogether by keeping the molecule suspended in solution, and differential detection cancels drift that is common to both traps, which is why it has become the default for base-pair-resolution work. Measuring the noise floor The noise of an instrument should be characterised with a method that shows how precision depends on averaging time. The Allan deviation does exactly this. For each averaging time τ, the data are divided into consecutive bins of length τ, the mean of each bin is computed, and the Allan variance is half the mean squared difference between adjacent bin means. Plotted against τ on logarithmic axes, the Allan deviation of a trapped bead falls at short times as Brownian noise is averaged away, reaches a minimum, and then rises as drift takes over. The minimum identifies the best achievable precision and the averaging time at which it is reached. Fabian Czerwinski, Andrew Richardson, and Lene Oddershede advocated the method for optical traps in 2009, and Bob Lansdorp and Omar Saleh extended it in 2012 as a calibration tool in its own right. A useful habit is to record an Allan deviation for a trapped bead, and preferably for a tethered bead under typical tension, at the start of each experimental day. If the minimum has moved to shorter times or higher values, something in the instrument has changed. The measurement takes a few minutes and has prevented many wasted days. The principle announced at the start of this chapter follows directly. An instrument that can resolve 1 ångström in one second of averaging may resolve only 1 nanometre in one millisecond. A molecular event that lasts ten milliseconds and moves a bead by 3 ångströms will be invisible to that instrument, not because the instrument is noisy but because the event is too brief for the noise to be averaged away. Resolution must always be quoted with its bandwidth, and claims about detecting small steps should be tested against simulated data with the same noise characteristics before they are believed. Hashtags: #OpticalTweezers #SingleMoleculeBiophysics #OpticalTrapping #SingleMoleculeMechanics #TrapStiffness #ForceSpectroscopy #BackFocalPlaneInterferometry #BeadCalibration #EquipartitionCalibration #PowerSpectrumCalibration #DragCalibration #MomentumCalibration #BrownianMotion #PiconewtonForces #NanometreDisplacement #DualTrapGeometry #ForceClamp #PositionClamp #ConstantVelocityPulling #MolecularTethers #SurfaceChemistry #HydrodynamicCorrections #NoiseFloorAnalysis #AllanDeviation #FutureOfSingleMoleculeBiophysics

Download the Book (PDF): Introduction Every experimental scientist eventually meets the same wall. The experiment is clear in the mind, the hypothesis is sharp, and the instrument that would answer the question either does not exist or costs more than the grant can bear. A motorized stage for a time-lapse microscope, a temperature-controlled chamber for a small organism, a syringe pump to deliver reagent at a slow and steady rate, a spectrophotometer that can live in a field station without mains power: each of these can be bought, and each can cost from several thousand to several tens of thousands of dollars. Most of that price is not the physics. It is the engineering overhead, the service contract, the certification of a general-purpose product for a broad market, and the margin a specialist supplier needs to survive in a small one. Over the past fifteen years a different path has matured. Desktop 3D printers now produce structural parts with a positional accuracy of a fraction of a millimetre for the cost of a few grams of plastic. Microcontroller boards costing less than a restaurant meal read sensors, drive motors, and keep time to microseconds. Single-board computers running full Linux can operate a camera, serve a web interface, and log data for months. Free computer-aided design tools, free circuit-design suites, and a global community that shares designs under open licences have turned what was once a machine-shop job into something a graduate student can do at a desk. The consequences for research are already visible. Joshua Pearce, whose 2012 paper in Science argued that free and open-source hardware could transform how scientists build research equipment, went on to document case after case in which open designs replaced commercial instruments at a small fraction of their cost. The OpenFlexure microscope, developed by Richard Bowman and colleagues, uses a single printed flexure mechanism to give sub-micron motorized positioning and has been built by laboratories and clinics around the world. The UC2 project from Jena packs optical components into printed cubes that snap together like a construction kit. Two peer-reviewed journals, HardwareX and the Journal of Open Hardware, now exist specifically so that an instrument design can be published, cited, and reproduced in the same way as a method or a dataset. What this book argues It would be easy to write a book that celebrates cheapness. That would be a mistake, because cheapness alone is a weak reason to build an instrument, and a dangerous one. A home-built pH logger that drifts by half a unit over a week is not a bargain; it is a source of wrong answers that look like right ones. The argument of this book is narrower and, I think, more useful: A home-built instrument is worth building when it is treated as an instrument rather than a gadget: specified before it is modelled, designed for the process that will make it, validated against a reference, documented so that someone else can rebuild it, and licensed so that they are allowed to. Done that way, open hardware delivers not only a large reduction in cost but something commercial equipment rarely offers, which is complete knowledge of how your measurement is made. The savings are real and often dramatic. But the deeper benefit is epistemic. When you have designed the optical path, chosen the sensor, written the sampling loop, and measured the drift, you know exactly what your instrument can and cannot tell you. Closed commercial systems often hide their signal processing behind firmware you cannot read. An open instrument can be inspected, audited, modified, and improved by anyone who doubts it. That is the same principle that makes open data and open code valuable in science, applied to the physical layer where measurements begin. Who this book is for The reader I have in mind is a working scientist, engineer, or technician: a doctoral student in cell biology who needs an incubator for a microscope stage, an environmental chemist who wants a dozen field loggers instead of one, a physics lecturer building teaching apparatus, a lab manager weighing whether to support a maker space, or a research-software engineer moving into hardware. No prior experience with 3D printing or electronics is assumed, but I do assume comfort with quantitative reasoning and a willingness to learn a little code. Where code appears, it is short, and it is there because it teaches something that prose cannot, such as how to timestamp samples properly or how a feedback loop behaves. This is not a catalogue of projects. Designs change quickly, and any list of specific builds would be out of date before it was printed. Instead, the book concentrates on the principles and practices that stay stable: how to turn a scientific need into a specification, how printed parts actually behave under load and heat, how analog-to-digital conversion limits what a sensor can report, how flexures and kinematic mounts give precision from imprecise materials, how to prove that an instrument measures what it claims to, and how to share the result so that it survives beyond the student who built it. How the book is organised The first chapter makes the case for building, and also the case against it. It sets out what open-source hardware means, where the economic evidence comes from, and how to recognise the projects that should be bought rather than built. Chapters 2 and 3 cover the mechanical layer. Chapter 2 is about computer-aided design for scientists: parametric modelling, the choice between code-based and graphical tools, and the habits that make a design adaptable and reproducible. Chapter 3 follows the model to the printer, covering materials, process physics, dimensional accuracy, and the specific failure modes, such as creep and warping, that matter when a printed part must hold an optic or a sample steady for hours. Chapters 4 and 5 move to electronics and computation. Chapter 4 treats microcontrollers as measurement devices, with attention to analog conversion, timing, sensor buses, and the everyday discipline of wiring and grounding. Chapter 5 addresses the single-board computer, the Raspberry Pi above all, as the brain of an instrument: running cameras, coordinating microcontrollers, storing data, and presenting an interface. Chapters 6 and 7 bring these layers together into working instruments. Chapter 6 is about optics: kinematic mounts, printed flexure stages, the OpenFlexure and UC2 systems, and the practical realities of stray light, thermal drift, and alignment. Chapter 7 covers motion and control: stepper motors, syringe pumps, and closed-loop temperature control, with a worked introduction to PID regulation. Chapter 8 is the chapter that separates instrumentation from tinkering. It covers calibration, uncertainty, comparison against reference instruments, and the documentation of performance that a reviewer, a collaborator, or your own future self will need in order to trust the data. Chapter 9 addresses licensing, documentation, and publication: the CERN Open Hardware Licence version 2 and its three variants, the OSHWA certification programme, the documentation expectations set out in DIN SPEC 3105, and how to publish a design in HardwareX or the Journal of Open Hardware so that it counts as a scholarly contribution. The conclusion draws out what follows for individual researchers, laboratories, and institutions if the argument of the book is right. A note on safety and responsibility Home-built equipment moves responsibility from a manufacturer to you. Mains electricity, lasers, heated elements, pressurised fluids, and ultraviolet light can all injure, and a laboratory's safety obligations do not relax because an instrument was cheap. Throughout the book, designs are kept to low-voltage direct current wherever possible, and the text flags the points at which a project needs a qualified electrician, a laser safety officer, or a formal risk assessment. Printed parts that contact biological material raise questions of sterilisation and contamination that are treated explicitly. None of this is meant to discourage building. It is meant to make sure that what you build is fit to sit on a laboratory bench alongside equipment that cost a hundred times more. The tools are cheap and the knowledge is freely shared. What remains scarce is the discipline to use them well. That discipline is what the chapters that follow try to supply. Chapter 1: The Case for Building, and When Not To Scientific instruments occupy a strange economic position. They are built for small markets, often a few thousand buyers worldwide, and so their development costs are spread across few units. They must be robust enough to survive careless users and general enough to serve many different experiments, which adds features most buyers never use. They carry regulatory and certification overheads, a sales and service network, and the risk premium of a company that may sell only a few hundred of a model before it becomes obsolete. The result is that a motorised microscope stage might sell for many thousands of dollars when its bill of materials, bought retail, would be a small fraction of that. None of this makes commercial suppliers villains. Their prices pay for real engineering, real support, and the assurance that the thing will work on the day it arrives. But the gap between the cost of the physics and the price of the product is precisely the space in which open-source hardware operates. What "open-source hardware" means The term has a specific meaning, and it is worth fixing it before going further. The Open Source Hardware Association (OSHWA) maintains the Open Source Hardware Definition, first published in 2011 and modelled on the Open Source Definition for software. In short, open-source hardware is hardware whose design is made publicly available so that anyone can study, modify, distribute, make, and sell the design or hardware based on it. The definition requires that the design files be released in the preferred format for making modifications, that the licence not discriminate against people, groups, or fields of endeavour, and that derived works be allowed. The phrase "preferred format for making modifications" does a great deal of work. A PDF drawing of a part, or an STL mesh exported from a CAD program, lets you reproduce the part but not easily change it. The preferred format is the native CAD source: the OpenSCAD script, the FreeCAD document, the KiCad project. An instrument released only as STL files and a photograph is shared, but it is not fully open in the sense the definition intends. Open-source hardware is not the same thing as do-it-yourself hardware. A student who builds a custom rig and never shares the design has built DIY hardware. It may be excellent, but it contributes nothing to anyone else and dies when the student graduates. Open hardware adds three things: complete documentation, a licence that grants explicit permissions, and public availability. Those additions are what let a design accumulate improvements from many users over time, which is where the long-run value lies. Nor is open-source hardware the same as cheap hardware. Some open designs, such as liquid-handling robots or light-sheet microscopes, still cost thousands of dollars to build because their components are intrinsically expensive. What they save is the markup, not the physics. Conversely, plenty of cheap equipment is completely closed. The economic evidence The strongest body of evidence on cost comes from Joshua Pearce, first at Michigan Technological University and later at Western University in Canada. His 2012 article in Science, "Building research equipment with free, open-source hardware," set out the argument that the combination of open-source microcontrollers and RepRap-derived 3D printers could let scientists fabricate much of their own equipment. His 2014 book Open-Source Lab: How to Build Your Own Hardware and Reduce Research Costs developed the case in detail, with worked designs and instructions. Pearce and his collaborators followed with specific designs whose savings they could quantify. A library of printable optical components, published in PLOS ONE in 2013 by Zhang, Anzalone, Faria, and Pearce, showed that mounts, filter wheels, and other optics hardware could be printed for a small percentage of catalogue prices. An open-source syringe pump library, published in the same journal in 2014 by Wijnen and colleagues, did the same for fluid delivery. In 2016 Pearce published a short analysis in Science and Public Policy on the return on investment when funding agencies pay for open hardware development: because a single design, once released, can be replicated by many laboratories, the return to the funder grows with every copy, in a way that a grant spent on a single commercial instrument cannot. His 2020 review in HardwareX, "Economic savings for scientific free and open source technology," surveyed a wide range of published open designs and found that savings relative to proprietary equivalents were typically large, very often exceeding eighty per cent, with further gains when designs were digitally replicated rather than rebuilt from scratch. It is worth being clear about what these numbers do and do not show. They compare the cost of materials for an open build against the list price of a commercial product with similar headline specifications. They generally do not include the builder's time, the cost of failed prints and false starts, or the value of the service and warranty that come with a commercial product. For a first build by a novice, time can dominate. For the tenth copy of a well-documented design, it largely disappears. The honest summary is that the savings are real, frequently very large, and strongly dependent on whether you are developing a design or reproducing one. A second strand of evidence is qualitative but just as important. In 2015, Tom Baden, Andre Maia Chagas, and colleagues published "Open Labware: 3-D printing your own lab equipment" in PLOS Biology, arguing that open, printable, Arduino-based tools would be especially transformative for laboratories in low-resource settings. Their follow-up, the FlyPi platform described in "The €100 lab" in 2017, combined a Raspberry Pi, a camera, printed parts, and simple electronics into a system for fluorescence imaging, optogenetic stimulation, and temperature control for small model organisms. A 2023 essay in PLOS Biology by Tobias Wenzel surveyed how open hardware had moved from a hobbyist trend to a genuine route to equipment access across the world. The pattern in this literature is that open designs are most transformative where commercial equipment is not merely expensive but unobtainable: where import duties, lack of local service, or currency constraints put it permanently out of reach. Beyond cost: the scientific arguments Cost is the argument that wins budget meetings, but it is not the only reason, and for many researchers it is not the most important one. The first scientific argument is transparency. A commercial instrument often performs signal processing you cannot see. A plate reader may apply smoothing, a spectrometer may apply a proprietary baseline correction, a camera may apply noise reduction in firmware. Usually these are sensible choices, but they are choices made by someone else, and they can interact badly with an unusual experiment. With an open instrument, every step from photons or volts to recorded numbers is available for inspection. When a result looks strange, you can trace it. The second is customisation. Commercial instruments are designed for the median experiment. If your experiment needs an unusual geometry, such as a microscope that looks up into a flow cell, a stage that holds a non-standard sample carrier, or a sensor array arranged around a plant stem, you may find nothing on the market that fits. Designing your own instrument means the tool fits the question rather than the reverse. The third is scale. Some experiments need many instruments rather than one good one. A study of behaviour across hundreds of flies, a network of soil sensors across a watershed, a parallel set of bioreactors: in each case the scientific value comes from replication, and the per-unit cost determines the size of the study. The ethoscope platform, described in PLOS Biology in 2017 by Geissmann and colleagues, is an example: an open, Raspberry Pi-based system for high-throughput monitoring of fly behaviour, designed to be deployed in large numbers. The fourth is reproducibility. If another lab wants to repeat your experiment and you used a commercial instrument that has since been discontinued, they may be unable to. If you used an open instrument with complete documentation, they can build it. This argument grows more compelling every year as research funders and journals press for reproducible methods. The fifth is education. Building an instrument teaches its principles in a way that operating one does not. A student who has assembled a spectrometer knows why stray light limits the measurable absorbance, because she has seen it happen. When you should not build An honest book on this subject has to say plainly that many instruments should be bought. The following considerations should push you toward a commercial product. When the measurement is regulated. Clinical diagnostics, food safety testing, and many environmental compliance measurements must be made on instruments that meet specific regulatory requirements. A home-built device may be perfectly accurate and still be unusable for these purposes because it lacks the required certification and quality system. Open designs can pass through regulatory processes, but that is a substantial project in its own right, not a side effect of printing some parts. When the physics is expensive and the markup is small. A high-quality scientific camera, a mass spectrometer, a cryostat, or a femtosecond laser derives most of its price from components that are genuinely costly to make. Building around such a component can still make sense, but you should not expect to reproduce the component itself. When the stakes of failure are high and the tolerance for downtime is low. A core facility serving many users needs an instrument that works every day and a service engineer who arrives when it does not. A research group's one-off experiment can tolerate a week of troubleshooting; a clinical workflow cannot. When the time cost exceeds the budget saving. A doctoral student has perhaps four years of research time. Spending six months developing an instrument that could have been bought for a modest sum is a poor trade unless the instrument is itself a contribution or enables something otherwise impossible. The right comparison is not the purchase price against the parts cost; it is the purchase price against parts plus the fully costed time of the builder. When a good open design already exists and someone sells it assembled. Many open instruments are sold as kits or finished products by small companies. Buying an assembled OpenFlexure microscope or an Opentrons liquid handler supports the project, saves your time, and still gives you the full benefits of openness. Building from scratch is not a virtue in itself. Counting the costs honestly A worked comparison shows how the arithmetic changes with circumstance. Suppose a lab needs a syringe pump for slow, steady infusion into a microfluidic device. A commercial research pump might be quoted at, say, two thousand dollars. An open design built from a stepper motor, a lead screw, a printed carriage, a microcontroller, and a driver board might need parts costing around one hundred dollars. On a parts-only comparison the saving looks like ninety-five per cent. Now add the other costs, still for this hypothetical case. The builder has never used a 3D printer, so the first week goes on learning slicing software and dealing with failed prints. A further week goes on wiring, firmware, and discovering that the chosen lead screw has enough backlash to make the flow pulse at low rates. A third week goes on calibration against a balance, measuring delivered mass over time. If the builder's time is costed at a modest research salary, three weeks may well exceed the price of the commercial pump. On the first unit, the open build loses. Build a second unit, and the picture shifts. The design is now known, the calibration procedure exists, and the second pump takes an afternoon. Build eight, for an experiment that needs parallel infusion into eight devices, and the open design wins decisively, because the commercial alternative is now sixteen thousand dollars and the open one is eight hundred dollars plus perhaps a week of work. Publish the design with its calibration data, and every other lab that builds it inherits the first builder's three weeks of learning for free. That last step is where the economics of open hardware become genuinely transformative, and it is why this book treats documentation and publication as part of the build rather than an afterthought. The general lesson is that open hardware favours multiplicity: many units, many users, many labs. For a single instrument used once, the case rests more on customisation and transparency than on money. A decision procedure Faced with a need, it helps to work through a short sequence of questions before opening any CAD software. Write down the measurement you need in quantitative terms: what quantity, over what range, with what resolution and accuracy, at what rate, for how long, in what environment. Search for existing open designs in HardwareX, the Journal of Open Hardware, the OSHWA certification directory, and the major repositories. Many problems have been solved already. Search for commercial equivalents and note their price, specifications, and whether they meet your quantitative requirements rather than just their headline ones. Estimate the build cost honestly: parts, printing, tools you do not already own, and your time at a realistic rate. Double the time estimate for a first build. Identify the reference against which you will validate the instrument. If there is no way to check it, you cannot trust it, and you should reconsider. Decide in advance how you will document and share the design. The fifth question is the one people skip, and it is the one that most often separates useful instruments from expensive toys. An instrument you cannot validate is not an instrument. The shape of a typical build Most open laboratory instruments share a common architecture, and recognising it early makes design much easier. There is a mechanical layer, usually printed parts combined with standard hardware such as screws, bearings, and extruded aluminium profile. There is a sensing and actuation layer: sensors that convert physical quantities to voltages or digital signals, and actuators such as motors, heaters, LEDs, and valves. There is a real-time control layer, usually a microcontroller, which reads sensors and drives actuators with precise timing. And there is often a supervisory layer, a single-board computer or a laptop, which handles user interface, heavy computation such as image processing, and long-term data storage. Separating these layers is good engineering. The microcontroller does not run an operating system and so can respond to events within microseconds, reliably, every time. The single-board computer runs Linux and so can do almost anything, but it cannot guarantee when it will do it. Putting timing-critical work on the microcontroller and everything else on the computer is the pattern that the rest of this book returns to repeatedly. A worked example makes this concrete. Suppose a lab needs to image the growth of seedlings on agar plates every ten minutes for a week, under controlled illumination, with plates kept at a constant temperature. The mechanical layer is a printed frame holding the plates at a fixed distance from a camera, with a light-tight enclosure. The sensing and actuation layer is a camera module, an array of LEDs, a temperature sensor, and a small heater. The real-time layer is a microcontroller that runs the temperature control loop and switches the LEDs. The supervisory layer is a Raspberry Pi that triggers the camera, tells the microcontroller when to switch lighting, saves images with timestamps, and exposes a simple web page showing the latest image and the temperature record. Every part of this system is well within the capability of a motivated researcher, and each is covered in the chapters that follow. Community and continuity One final consideration shapes everything else. Instruments built by a single person tend to die when that person leaves. Open instruments that attract a community of users tend to survive and improve. The Gathering for Open Science Hardware (GOSH), which has held meetings since 2016 and produced a manifesto setting out principles for the field, is the most visible expression of that community in science. Around specific projects, such as OpenFlexure and UC2, there are forums, repositories with issue trackers, and small businesses selling kits. When deciding whether to build, it is worth asking not only "Can I make this?" but "Who will maintain it?" A design that joins an existing community project, or that is documented well enough to start one, has a future. A design that exists only in the builder's head and a folder of unlabelled STL files does not. The rest of this book is written in the conviction that the first kind of instrument is worth the extra effort. Chapter 2: From Requirement to Model: CAD for Scientists A 3D printer will faithfully produce whatever you give it, including your mistakes. The quality of a printed instrument is set almost entirely before the printer starts, in the translation of a scientific need into a geometric model. That translation is the subject of this chapter. It covers how to write a specification a model can be tested against, how to choose among the main open and freely available CAD tools, why parametric design matters so much for scientific hardware, and the design habits that make parts printable, adjustable, and shareable. Start with a specification, not a shape The most common mistake new designers make is to open a CAD program and start drawing. The shape feels like the design, so it seems natural to begin there. But the shape is the answer to a set of requirements, and if those requirements have not been written down, there is no way to know whether the shape is right. A useful specification for a mechanical part in an instrument answers questions of the following kind. What must this part hold, and in what position relative to what else? Which dimensions are critical, and to what tolerance? What loads will it carry, for how long, and at what temperature? What will it touch: optics, samples, solvents, living cells, skin? How often will it be assembled and disassembled? What standard components must it interface with, such as a 25 mm lens, an M3 screw, a 1-inch optical post, a Raspberry Pi camera board, or a standard microscope slide? Consider a holder for a standard 10 mm path-length cuvette in a simple absorbance instrument. The external dimensions of such cuvettes are nominally 12.5 mm square, but they vary slightly between manufacturers, and a printed pocket will not come out exactly as modelled. The critical requirement is not the pocket size itself but the position of the optical path: the light beam must pass through the cuvette at a known height, perpendicular to its faces, without clipping the walls. So the specification might read: accept cuvettes of 12.5 mm nominal outer width with a repeatable fit; locate the cuvette against two reference faces so that it sits in the same position each time; place the beam axis at a stated height above the cuvette base; provide apertures on the beam axis of a stated diameter; block all other light. Once that is written, the geometry nearly designs itself, and the specification tells you what to measure when the part is printed. The habit of specifying first pays off most in the questions it forces. "Locate against two reference faces" is a design principle, not an afterthought: a part that relies on a snug fit on all four sides will locate differently each time depending on which side touches, whereas a part that pushes the cuvette into a corner with a light spring or a flexure will locate the same way every time. That idea, known in precision engineering as exact constraint, recurs throughout the book. Choosing a tool There is no single correct CAD program for scientific hardware, but there are real differences in how the main options suit open, reproducible work. Table 1 compares the tools most often used for open laboratory hardware. Table 1. Commonly used CAD tools for open scientific hardware. Tool Modelling style Licence and access Suits OpenSCAD Code-based solid modelling Free and open source Simple parametric parts, version control FreeCAD Graphical parametric, sketch and feature Free and open source General mechanical design, assemblies CadQuery / build123d Python code on a professional kernel Free and open source Complex parametric parts in code Onshape Graphical, browser-based Proprietary; free plan for public documents Collaborative design Fusion Graphical parametric Proprietary; restricted free personal licence Complex surfaces, integrated CAM KiCad Schematic and circuit-board layout Free and open source Electronics design OpenSCAD describes geometry as a program. You write code that creates primitive solids such as cubes, cylinders, and spheres, transforms them, and combines them with union, difference, and intersection operations. There is no clicking and dragging. This sounds austere, and for organic shapes it is, but for laboratory hardware it has substantial advantages. The design is a plain text file, so it can be stored in Git, compared line by line between versions, and reviewed like any other code. Every dimension is a named variable, so changing a lens diameter from 25 mm to 12.7 mm is a single edit. The OpenFlexure microscope was for years developed largely in OpenSCAD, which is one reason it has been so readily adapted. FreeCAD is a full graphical parametric modeller. You draw constrained two-dimensional sketches, extrude or revolve them into solids, and add features such as holes, fillets, and chamfers. Every feature records its parameters, so the model can be regenerated after a change. FreeCAD reached its version 1.0 release in late 2024, which addressed some long-standing stability problems in how features reference the geometry they are built on. It also includes workbenches for assemblies, technical drawings, and finite-element analysis. CadQuery and its close relative build123d are Python libraries that sit on top of the same OpenCASCADE geometry kernel that FreeCAD uses. They combine the text-based, version-controllable nature of OpenSCAD with the richer operations of a professional kernel, such as true fillets, chamfers on selected edges, and export to STEP files that other CAD programs can edit. For a scientist already fluent in Python, they are often the most natural choice. Onshape and Fusion are proprietary but widely used. Onshape runs in a browser and stores designs in the cloud, which makes collaboration easy; its free plan requires that documents be public, which suits open hardware. Fusion is powerful but its free licence is restricted and its terms have changed several times. Designs made in proprietary tools can still be released openly, but anyone who wants to modify the native files needs access to the same tool. For long-lived open hardware, a design that can be edited only in a proprietary cloud service is a risk: if the service changes its terms, the preferred form for modification may vanish. A reasonable compromise is to design in whatever tool you are fluent in and to publish neutral STEP files alongside the native source. KiCad is the standard open-source suite for electronics: schematic capture, printed-circuit-board layout, and generation of the files a board manufacturer needs. It is covered further in Chapter 4, but it belongs in any discussion of design tools because many instruments need at least a simple custom board to replace fragile breadboard wiring. Parametric design as scientific practice Parametric design means that the model is driven by named parameters and the relationships between them, rather than by fixed numbers typed into individual features. It is standard practice in engineering, but in scientific hardware it is more than convenience. It is the mechanism by which a design becomes adaptable by other people. Here is a short OpenSCAD example: a mount that holds a round lens or filter of any diameter in a printed ring, with a clamping screw. // Parametric lens ring mount lens_d = 25.4; // lens diameter, mm lens_t = 5.0; // lens edge thickness, mm clearance = 0.3; // radial allowance for print shrinkage and fit wall = 3.0; // ring wall thickness lip = 1.0; // retaining lip width screw_d = 3.2; // clearance hole for M3 clamp screw bore = lens_d + 2*clearance; outer = bore + 2*wall; difference() { cylinder(d = outer, h = lens_t + 2, $fn = 120); // lens pocket translate([0, 0, 1]) cylinder(d = bore, h = lens_t + 2, $fn = 120); // clear aperture, leaving a lip that the lens rests on translate([0, 0, -1]) cylinder(d = lens_d - 2*lip, h = 3, $fn = 120); // radial clamp screw hole through the wall translate([0, 0, 1 + lens_t/2]) rotate([0, 90, 0]) cylinder(d = screw_d, h = outer, $fn = 30); } Every number that matters has a name and a comment. Someone with a 12 mm lens changes one line. Someone whose printer produces holes slightly undersized changes the clearance. The design carries its own reasoning with it. Contrast this with a mesh file downloaded from a repository: it may fit a 25.4 mm lens on the author's printer, but it tells you nothing about why, and adapting it means reverse-engineering the geometry. The same idea in CadQuery looks like ordinary Python. Here is a small baseplate with four counterbored holes for M3 screws on a 50 by 30 mm pattern, with rounded vertical corners, exported as a STEP file that any other CAD program can open: import cadquery as cq L, W, T = 60, 40, 6 # plate size, mm PX, PY = 50, 30 # hole pattern, mm plate = ( cq.Workplane("XY").box(L, W, T) .edges("|Z").fillet(3) # round the vertical corners .faces(">Z").workplane() .rect(PX, PY, forConstruction=True).vertices() .cboreHole(3.4, 6.5, 3.5) # M3 clearance, head counterbore ) cq.exporters.export(plate, "base_plate.step") Two features distinguish this from the OpenSCAD version. The fillet is a true rounded edge computed by the geometry kernel, which OpenSCAD can only approximate. And the output is a STEP file, which preserves exact geometry rather than the triangulated approximation of an STL mesh, so a colleague using FreeCAD, Onshape, or a commercial package can import the plate and continue working on it. The price is a steeper learning curve and a heavier software installation. For a scientist who already writes Python for data analysis, that price is usually small. The clearance parameter deserves attention. Fused-filament printers tend to print holes slightly smaller than modelled, because of how the extruded bead spreads and how curves are approximated by straight segments. The size of the error depends on the printer, the material, and the slicer settings, and it is typically a few tenths of a millimetre. Rather than hard-coding an allowance that suits one printer, good open designs expose it as a parameter and provide a small test print so that each builder can measure their own printer and set it correctly. Chapter 3 returns to this. A few further habits make parametric designs robust: • Derive dependent dimensions from independent ones, as bore and outer are derived above, so that changing one parameter updates everything consistently. • Keep interface dimensions, such as screw spacings that must match another part, in a shared file that both parts include, so they cannot drift apart. • Model standard parts, such as screws, nuts, and bearings, as simple placeholders so that clearance can be checked visually. • Use a coordinate origin that has physical meaning, such as the optical axis or the sample plane, so that other people can reason about positions. Designing for manufacture and reuse A part can be geometrically correct and still unprintable, or printable only with support material that ruins critical surfaces. Designing for additive manufacturing means understanding what the process can and cannot do and shaping the part to suit it. Fused-filament printing builds parts one layer at a time from the bottom up. Each layer must rest on the one below it. Surfaces that overhang by more than roughly 45 degrees from vertical will sag unless supported, and horizontal bridges across empty space can span only limited distances before drooping. Holes printed with their axis horizontal come out slightly teardrop-shaped at the top. Holes with their axis vertical come out round but tend to be undersized. Parts are strongest within a layer and weakest between layers, because interlayer bonding is less complete than bonding along an extruded line. Several design responses follow directly. Orient parts so that the critical surfaces are either on the build plate, where they are flat and accurate, or vertical walls, where they are precise. Replace overhangs with 45-degree chamfers wherever possible. Model horizontal holes with a small pointed top, the "teardrop" profile, so that they print without support and can accept a round shaft. Arrange load paths so that stresses run along layers rather than trying to peel them apart; a thin bracket printed upright will snap along a layer line under a load that the same bracket printed flat would easily bear. Threads illustrate the trade-offs well. Printed threads for large diameters work acceptably, but small screw threads printed directly are weak and imprecise. Better options are to print a hole and let a self-tapping screw cut its own thread, to capture a hexagonal nut in a printed pocket, or to press heat-set threaded inserts into the plastic with a soldering iron. Heat-set brass inserts give durable metal threads that tolerate repeated assembly, and they are now common in open instrument designs. The design needs a hole of the diameter the insert manufacturer specifies, plus a little extra depth so that displaced plastic has somewhere to go. Designing for the process also means designing for assembly. Parts that must align should include registration features such as pins, keys, or dovetails rather than relying on the builder to line things up by eye. Screws should be accessible with ordinary tools. Cable routes should be planned rather than improvised. The best open designs also include assembly instructions that are generated from, or at least checked against, the model, and some projects have built tools to automate this. Designing around standards A printed part rarely lives alone. It bolts to an optical table, clips onto an aluminium frame, holds a commercial lens tube, or cradles a circuit board. The single most effective way to make a printed instrument useful to other people is to design its interfaces around standards that already exist in laboratories, so that printed parts and bought parts can be mixed freely. Optical breadboards and tables use a grid of tapped holes, either 1/4-20 threads on a 1-inch pitch in imperial systems or M6 threads on a 25 mm pitch in metric ones. A printed base that places its mounting slots on that pitch can sit anywhere on any table. Optical posts of half-inch or 12 mm diameter are ubiquitous, and a printed post clamp is one of the most useful parts a lab can have. Cage systems, in which optics are held in plates threaded onto four parallel rods, provide an excellent skeleton for compact instruments: the widely used 30 mm cage standard puts 6 mm rods on a 30 mm square, and a printed plate with four precisely sized holes on that square can hold any custom component coaxially with bought lenses and filters. Aluminium extrusion profiles, most commonly 20 mm square with a slot on each face, give rigid frames for enclosures and larger instruments, and printed brackets can attach to them with standard T-nuts. Electronics have their own conventions. Raspberry Pi boards and their camera modules have published mounting-hole positions, as do the common Arduino form factors. Designing mounts around these published drawings, rather than measuring a board with calipers, avoids errors and makes the design correct for everyone who owns the same board. Using standards has a further, less obvious benefit. It lets a design degrade gracefully. If a printed part proves not stiff enough, a builder can substitute a machined or bought part with the same interface without redesigning anything else. The UC2 system, discussed in Chapter 6, is built on exactly this idea: a single cube standard into which any component can be inserted, printed or not. Version control and file hygiene Scientific instruments evolve. A mount is revised to take a thicker filter; a stage gets a stiffer flexure; a new camera module requires a different bracket. Unless the design files are managed carefully, it quickly becomes impossible to say which version of which part was in the instrument that produced a particular dataset. The solution is borrowed from software. Store the design source in a version-control system such as Git, hosted on a platform such as GitHub, GitLab, or Codeberg. Commit changes with messages that explain why. Tag releases with version numbers, and record in your lab notebook or data files which release of the instrument generated each dataset. Code-based CAD tools fit this workflow naturally because their files are plain text. Graphical tools produce binary or compressed files that Git can store but cannot usefully compare, so for these it helps to also export neutral formats and to keep a written change log. A clean repository for an open instrument typically contains the native CAD source, exported STL files for printing, STEP files for interoperability, the electronics project if any, firmware and software, a bill of materials, assembly instructions, calibration procedures and data, and a licence file. That last item is the subject of Chapter 9, but it should be decided early: it is far easier to choose a licence at the start than to trace every contributor later to obtain their agreement. From model to measurable part The final discipline of design is to make the model testable. Every critical dimension in the specification should correspond to something that can be measured on the printed part with calipers, a gauge pin, a microscope, or a test fit. If the specification says the beam axis sits 15 mm above the cuvette base, the design should include a reference surface from which that height can be measured. If the design relies on a particular clearance, the repository should include a small test coupon that prints quickly and lets the builder check that clearance before committing hours of printer time to the full part. This is the point at which design stops being an act of imagination and becomes an engineering process with feedback. The model makes a prediction; the printed part tests it; the discrepancy updates the model. Chapter 3 follows that loop into the printer itself. Chapter 3: Printing Parts That Behave A printed part in an instrument has a job. It must hold something in a known position, keep it there as the room warms and cools, survive the chemicals and handling of a working lab, and not introduce artefacts into the measurement. Whether it succeeds depends on the printing process, the material, and the settings far more than on how the part looks. This chapter explains the physics of the two most common printing processes, how to choose materials for laboratory use, how to get dimensions right, and the failure modes that matter most for scientific hardware. Two processes, two sets of trade-offs Fused filament fabrication (FFF), also called fused deposition modelling, is the process most people mean by 3D printing. A thermoplastic filament, usually 1.75 mm in diameter, is pushed into a heated nozzle, melted, and extruded as a thin bead, typically around 0.4 mm wide, which the printer lays down along a path computed by slicing software. Layer after layer builds the part. FFF printers are inexpensive, robust, and can use a wide range of engineering thermoplastics. Their parts are anisotropic, stronger along layers than across them, and have a visibly stepped surface on curved and sloping faces. Dimensional accuracy on a well-tuned machine is usually within a couple of tenths of a millimetre on features of ordinary size. Vat photopolymerisation, including stereolithography (SLA) and masked stereolithography using an LCD screen (MSLA), builds parts by curing a liquid resin with ultraviolet light, one thin layer at a time. It produces much finer detail and smoother surfaces than FFF, with nearly isotropic properties, and it can make small channels and fine features that FFF cannot. Its drawbacks are that most resins are more brittle than engineering thermoplastics, that uncured resin is a skin sensitiser and must be handled with gloves and good ventilation, that printed parts must be washed and post-cured, and that some cured resins leach compounds that can harm living samples. For structural parts of instruments, such as frames, mounts, stages, and enclosures, FFF is usually the right choice: its materials are tougher, cheaper, and more predictable under long-term load. For small, detailed parts, such as microfluidic moulds, fine gears, or components that must be optically smooth, resin printing may be better. Many labs that build instruments end up with one of each. Choosing materials The material question is where laboratory hardware differs most from hobby printing. Heat, solvents, sterilisation, sustained load, and optical behaviour all impose constraints. Table 2 summarises the common choices in qualitative terms. Table 2. Common printing materials and their suitability for laboratory parts. Material Heat tolerance Stiffness and creep Chemical resistance Typical lab role PLA Low; softens near 55-60 °C Stiff but creeps under load Poor to moderate Prototypes, room-temperature mounts PETG Moderate Tougher, slightly flexible Good to many aqueous media General structural parts ABS / ASA Higher Tough Attacked by some solvents Warmer environments, enclosures Polycarbonate High Very tough, stiff Moderate Heated stages, demanding loads Nylon (PA) High Tough, can absorb water Good to oils and many solvents Wear parts, gears, flexures Photopolymer resin Varies widely Often brittle Varies Fine detail, moulds, small parts PLA (polylactic acid) is the easiest material to print and the most common. It is stiff, dimensionally accurate, and prints with little warping. Its great weakness is heat: its glass transition lies around 55 to 60 °C, so a PLA part left on a warm instrument, in a car in summer, or inside an incubator enclosure can deform. Even well below that temperature, PLA creeps under sustained stress, which matters for any part that clamps an optic with a screw. PLA is excellent for prototypes and for room-temperature parts under light load. It is also the material in which the OpenFlexure microscope is usually printed; its flexures work within PLA's limits because the stresses in them are kept low by design. PETG (polyethylene terephthalate glycol-modified) is somewhat harder to print well but tougher, less brittle, and more heat-tolerant than PLA. It resists many aqueous solutions and is a good default for structural lab parts that may see some warmth or moisture. ABS and its weather-resistant relative ASA tolerate higher temperatures but shrink as they cool and tend to warp and crack unless printed in an enclosed, heated chamber. They also emit more fumes during printing. Their advantage is heat tolerance; their disadvantage, apart from printability, is that some organic solvents, such as acetone, dissolve them. Polycarbonate and nylon are engineering materials suited to parts that must survive heat, sustained load, or wear. Both need high nozzle temperatures, and nylon absorbs water from the air, which must be dried out of the filament before printing and which affects its mechanical properties in use. Fibre-filled versions of these and other polymers, containing chopped carbon or glass fibre, are much stiffer and more dimensionally stable, which is attractive for instrument frames, but they are abrasive and require hardened nozzles. Two material properties deserve special attention in scientific hardware. Optical behaviour. Plastics fluoresce. Some pigments and additives fluoresce strongly under blue or ultraviolet excitation, which can add background in a fluorescence instrument. Black filaments are generally preferable for optical enclosures and light baffles, but "black" in visible light does not guarantee opacity in the near infrared: depending on the pigment, some black plastics transmit appreciable infrared light. If an instrument uses infrared illumination or a camera sensitive to it, test the material by holding a printed sample between the source and the detector before relying on it. Thin walls of any plastic pass some light; a wall thickness of a couple of millimetres, or a coat of matt black paint on the inside, is a sensible precaution. Biological compatibility. Printed parts that contact cells, tissues, or organisms raise two issues: whether the material or its additives are toxic, and whether the part can be sterilised. Oskui and colleagues, writing in Environmental Science & Technology Letters in 2016, showed that parts made by some resin printers were markedly toxic to zebrafish embryos and that post-print treatment with ultraviolet light substantially reduced the toxicity. The general lesson extends beyond that study: test any printed material with your own biological system before using it in an experiment, and do not assume that a material described as "biocompatible" in a general sense is inert in yours. As for sterilisation, most common filaments cannot survive a standard autoclave cycle at 121 °C. PLA and PETG will deform. Low-temperature methods, such as wiping or immersion in 70 per cent ethanol, ultraviolet exposure, hydrogen peroxide vapour, or ethylene oxide where available, are more appropriate, and each should be validated for the purpose. The layered surface of an FFF part has crevices that can harbour contaminants, which is another reason to keep printed parts outside the sterile zone where possible and let them hold commercial sterile consumables instead. Getting dimensions right Printed parts rarely come out exactly as modelled. The discrepancies come from several sources, and knowing which is which makes them easy to correct. The first source is mechanical calibration. If the printer's motion system is not calibrated, all dimensions will be scaled incorrectly. This is checked by printing a simple test cube or bar of known length and measuring it, and corrected in the printer's firmware settings. On most modern printers it is correct out of the box. The second is extrusion calibration. If the printer extrudes slightly too much plastic, walls get thicker and holes get smaller; too little and parts are weak with gaps between lines. The fix is to calibrate the extrusion rate, often called flow or extrusion multiplier, for each filament, by printing a single-walled test object and measuring its wall thickness against the intended line width. The third is systematic process error that remains even on a well-calibrated machine: holes print undersized, and small features are distorted by the nozzle's finite width and the way the slicer approximates curves. This is where a test coupon earns its keep. A worked procedure makes this concrete. Model a flat plate with a row of vertical holes of nominal diameter 3, 5, 8, 12, and 20 mm. Print it with the same material and settings that the real part will use. Measure each hole with a set of gauge pins or the inside jaws of calipers. Suppose, in a hypothetical case, each hole measures about 0.25 mm smaller than modelled, fairly consistently across sizes. Then the right correction is a constant radial allowance, and the clearance parameter in a design like the lens mount of Chapter 2 should be set accordingly. If, instead, the error grows with diameter, the problem is scaling, and the machine calibration needs attention. If small holes are much more undersized than large ones, the slicer's approximation of small circles or the flow at short segments is involved, and small holes are better drilled or reamed to size after printing. That last option is often the best. For any hole that must be accurate, such as a bearing seat, a shaft bore, or a dowel-pin location, it is good practice to print it slightly undersized and then finish it with a drill or reamer of the correct size. Printing gets the hole in the right place; the reamer gets it the right size. Similarly, critical flat faces can be printed on the build plate, where they are flattest, or lightly sanded on a sheet of abrasive laid on glass. Settings that matter for strength and stability Slicing software exposes dozens of settings, but a handful dominate the properties of a functional part. Walls, or perimeters, are the outer shells of each layer. For strength and stiffness, more walls matter more than more infill. Three or four perimeters of a 0.4 mm nozzle give a solid skin that resists bending and allows holes to be drilled or tapped without breaking into infill. Infill fills the interior with a lattice. Denser infill increases stiffness and mass. For most instrument parts a moderate infill with a pattern that is strong in all directions, such as gyroid or cubic, is adequate. Parts that must be stiff and thermally stable, such as an optical baseplate, benefit from being printed solid or nearly so, because voids reduce stiffness and allow the part to flex. Layer height trades surface finish and vertical resolution against print time. Thinner layers give smoother slopes and finer vertical detail. For strength, layer height matters less than good interlayer bonding, which is controlled mostly by temperature and cooling. Orientation is the most important setting of all, and it is chosen in the slicer rather than the CAD program. As Chapter 2 noted, a part is weakest between layers, so orientation should put the principal stresses along layers. A flexure that must bend repeatedly should be printed so that its thin section is composed of continuous extruded lines along its length, not stacked layers across it. Temperature and cooling control interlayer bonding. Printing slightly hotter and with less part cooling fan improves bonding at the expense of surface detail on overhangs. For structural parts, stronger bonding is usually worth it. Failure modes and good practice Hobby printing worries about whether a part looks right. Instrument building must worry about how it behaves over time. Four failure modes deserve particular attention. Creep. Polymers deform slowly under sustained stress, even at room temperature, and more quickly as they warm. A screw clamping a mirror in a printed mount gradually loses tension as the plastic under it flows. A printed bracket carrying a camera slowly sags. The remedies are to keep stresses low by using larger contact areas, to use metal for parts that carry clamping loads (heat-set inserts, washers, metal springs), to preload with a spring rather than a rigid screw so that creep is taken up rather than lost, and to choose materials with better creep resistance for warm environments. Thermal expansion. Plastics expand with temperature considerably more than metals or glass. A printed part holding an optic a certain distance from a sensor will change that distance as the room temperature changes, and a focus that was sharp in the morning may be soft by afternoon. The effect is small per degree but meaningful at microscope scales. The remedies are to keep critical optical paths short, to design so that expansion moves things along the optical axis symmetrically rather than tilting them, to allow the instrument to reach thermal equilibrium before measuring, and, for the most demanding work, to use autofocus or active compensation. Chapter 6 discusses these strategies in the context of printed microscopes. Moisture. Some materials, nylon especially, absorb water, which changes their dimensions and stiffness. Parts used in humid environments or immersed in liquid may swell. Filaments themselves absorb water in storage, which degrades print quality; keep them sealed with desiccant and dry them before use if necessary. Fatigue and cracking. Parts that flex repeatedly, such as snap fits, flexures, and clips, eventually crack if the strain in them is too high. The rule is to design flexing parts so that their peak strain stays well within the material's elastic range. Longer, thinner flexures bend with less strain for the same deflection than short, thick ones. Hybrid construction: knowing when not to print One of the marks of an experienced builder is knowing which parts should not be printed at all. Printing is superb for complex, low-load geometry: brackets that hold components in awkward positions, enclosures with integrated cable routes, adapters between incompatible standards, jigs for assembly and alignment. It is poor for long, stiff members, for surfaces that must stay flat to a few micrometres, for bearings, and for anything that carries high sustained loads at elevated temperature. The most robust open instruments are therefore hybrids. A frame of aluminium extrusion provides stiffness over long spans; printed corner brackets and mounts attach components to it. Precision steel rods and linear bearings provide straight motion; printed carriages hold them. A machined or bought aluminium plate forms the baseplate of an optical path; printed mounts sit on it. Steel dowel pins pressed into reamed holes provide accurate registration between printed parts that must be taken apart and reassembled. Springs, magnets, ball bearings, and hardened steel balls, all cheap and standard, provide the precise contacts that plastic cannot. A useful rule is to let printed plastic define the topology of the instrument, where things are and how they connect, and to let metal and glass define the precision, the surfaces and contacts that set critical positions. Kinematic mounts, discussed in Chapter 6, are the clearest example: a printed body locates a mirror through three steel balls sitting in grooves or against flats, and it is the steel contacts, not the plastic, that make the mount repeatable. Recording what you printed Because material, settings, and orientation all change a printed part's behaviour, they belong in the instrument's documentation as much as the geometry does. A part printed in PETG at four perimeters and solid infill is not the same component as the same model printed in PLA at two perimeters and sparse infill, even though the files are identical. A builder who reproduces your instrument with the wrong settings may get a weaker, less stable version and conclude that the design is poor. Good practice is to record, for each printed part, the recommended material, nozzle size, layer height, number of perimeters, infill density and pattern, orientation on the build plate, and whether supports are needed. Many projects put this information in a table in the assembly instructions. It also helps to export and publish the slicer project or profile, though these formats change between slicer versions and should not be the only record. Within a lab, labelling printed parts with a small embossed version number, modelled into the part itself, makes it possible months later to tell which revision of a mount is installed in which instrument, which in turn makes it possible to connect a change in data quality to a change in hardware. Safety and the printing environment Printers themselves pose modest but real hazards. Nozzles and heated beds are hot enough to burn. FFF printers emit ultrafine particles and volatile organic compounds during printing; Stephens and colleagues reported substantial ultrafine particle emissions from desktop printers in Atmospheric Environment in 2013, and later work has confirmed that emissions vary with material and temperature, with ABS generally emitting more than PLA. Printers used routinely in a lab should sit in a ventilated space or an enclosure with filtration, not on an office desk beside people working for hours. Resin printers call for nitrile gloves, eye protection, and ventilation, and waste resin and washing solvent must be cured or disposed of as chemical waste according to local rules. These precautions are not onerous, and they are the same kind of precautions a lab takes with any piece of equipment. Treated with ordinary care, a well-tuned printer becomes a reliable machine for producing reliable parts, and that is the foundation on which everything else in this book is built. Hashtags: #OpenSourceHardwareInScience #OpenSourceHardware #ScientificInstrumentation #LowCostInstrumentation #ThreeDPrinting #Microcontrollers #SingleBoardComputers #Arduino #RaspberryPi #OpenFlexureMicroscope #UC2Microscopy #ParametricDesign #OpenSCAD #FreeCAD #CadQuery #KiCad #ScientificHardwareDesign #HardwareValidation #InstrumentCalibration #MeasurementUncertainty #OpenHardwareLicensing #CERNOpenHardwareLicence #OSHWA #ReproducibleInstrumentation #FutureOfOpenScientificHardware

Download the Book (PDF): Introduction A conventional environmental laboratory is very good at answering one kind of question: how much of this compound is in that sample? A method is validated for a list of analytes, calibration standards are bought, a triple-quadrupole instrument is tuned to watch for a handful of precursor-to-fragment transitions per compound, and the answer comes back in nanograms per litre with an uncertainty attached. That machinery underpins drinking-water compliance, pesticide residue monitoring and most regulatory chemistry in the world. Its weakness is built into its strength. A targeted method sees only what it was told to look for. Anything else in the sample, however abundant and however toxic, passes through the detector without leaving a trace in the report. The chemical universe that reaches rivers, groundwater, house dust and the products on a supermarket shelf is vastly larger than any target list. Registries of commercial chemicals run to hundreds of thousands of substances, and each of those substances can be accompanied by impurities, reaction by-products and transformation products formed in treatment plants, in sunlight, in soil and inside living organisms. Most of these have never been measured anywhere. Some of them matter a great deal. The clearest recent demonstration came from the Pacific Northwest of the United States, where coho salmon returning to urban creeks had been dying after rainstorms for decades without an explanation. Tian and colleagues, publishing in Science in 2021, traced the mortality to a previously unrecognised compound, a quinone formed when the tyre antioxidant 6PPD reacts with ozone. No targeted method could have found it, because nobody knew it existed. It was found by fractionating tyre-wear leachate, following the toxicity, and then interrogating the toxic fractions with high-resolution mass spectrometry until a formula, then a structure, then a confirmed identity emerged. That is non-targeted screening in its fullest form, and it is the subject of this book. The term covers a family of approaches that share one idea: acquire as much accurately measured mass-spectral information from a sample as the instrument allows, and decide afterwards what to look for. Some of that looking is guided by lists of suspected compounds. Some of it starts from the data themselves, with no hypothesis about identity at all. Either way the output is not a concentration table but a set of claims of the form "this signal is probably that compound", each with a stated degree of confidence. The argument of this book The central claim of the chapters that follow is simple to state and surprisingly easy to lose sight of in practice. An identification from high-resolution mass spectrometry is an argument assembled from independent lines of evidence, and its scientific value lies in reporting honestly how far that argument has got. Accurate mass narrows the possibilities to a handful of molecular formulas. The isotope pattern, and at high enough resolution the isotope fine structure, narrows them further and reveals elements such as chlorine, bromine and sulfur. Mass defect relationships place the unknown in a family of related compounds. Fragmentation spectra constrain the structure. Retention behaviour, database context and, finally, comparison with an authentic reference standard close the case. Each step removes candidates; none of them on its own proves anything. Laboratories get into trouble when they forget this. A software package matches a peak to a name in a database of a million compounds, a report lists the name, and the name acquires a life of its own in a slide deck or a press release. The literature contains many examples of "detections" that were really exact-mass coincidences. The discipline has responded by developing a shared vocabulary for confidence, most influentially the five-level scale proposed by Emma Schymanski and colleagues in 2014, and by building community resources, reporting tools and interlaboratory trials that make claims comparable between laboratories. Understanding those tools is as important as understanding the instruments. Who this book is for The intended reader is a scientist or technically literate professional who needs to understand, commission, perform or evaluate non-targeted screening: an environmental chemist moving from targeted work into high-resolution methods, a regulator or water utility scientist reading non-targeted reports, a product-safety specialist worried about what migrates out of packaging and textiles, a toxicologist who wants to know how much weight a tentative identification can bear. No prior experience with high-resolution instruments is assumed, but a working familiarity with chemical formulas and the basic idea of a mass spectrum will help. The book does not try to be an instrument manual. Vendors change their products every few years, and software moves faster still. What stays constant is the arithmetic of exact mass, the physics of isotopes, the logic of homologous series and the reasoning that connects a spectrum to a structure. Those are treated in depth, with worked numbers, so that a reader can check a claimed identification by hand and see where it is strong and where it is weak. How the book is organised Chapter 1 sets out the three modes of screening, targeted, suspect and non-targeted, and explains why environmental water and consumer goods present such rich and difficult problems. Chapter 2 explains what high-resolution instruments actually measure, with particular attention to the difference between resolving power and mass accuracy and to the arithmetic of parts per million. Chapter 3 follows a sample from the field or the product through extraction, quality control and data processing to the list of chromatographic features that everything else depends on. Chapter 4 covers suspect screening: where suspect lists come from, how the NORMAN Suspect List Exchange, the US EPA CompTox Chemicals Dashboard and PubChemLite are used, and why matching on exact mass alone produces so many false positives. Chapter 5 is about determining a molecular formula, and in particular about isotope patterns and isotope fine structure, including the mass differences that allow a sulfur atom to be distinguished from two carbon-13 atoms and the resolving power needed to do it. Chapter 6 explains Kendrick mass defect analysis entirely through worked arithmetic, showing how a change of mass scale makes whole families of per- and polyfluoroalkyl substances, surfactants and polymers line up. Chapter 7 addresses structure elucidation from tandem mass spectra, from library matching through to in silico fragmentation tools such as MetFrag and SIRIUS. Chapter 8 turns to confidence and reporting: the Schymanski levels, their extension for fluorinated compounds, the study reporting framework developed by the Benchmarking and Publications for Non-Targeted Analysis working group, and the unsolved problem of estimating concentrations for compounds with no standard. Chapter 9 brings everything together in two extended, hypothetical case studies, one in river water and one in a consumer product, and discusses retrospective screening of archived data. The conclusion considers what follows from treating identification as graded evidence rather than as a yes-or-no answer. A note on examples and numbers The exact masses used in worked examples throughout the book are calculated from standard monoisotopic atomic masses, with the mass of the electron taken into account for ions. Readers are encouraged to reproduce them; the arithmetic is the best way to develop an instinct for what a given mass error means. Where scenarios describe a laboratory, a river or a product, they are hypothetical and are presented as such. Where the book refers to published work, it is to papers, databases and guidance documents that exist and can be consulted, and they are collected in the Notes and Further Reading at the end. Non-targeted screening is still a young field by the standards of analytical chemistry. It lacks the settled validation procedures of targeted analysis, and its results are harder to communicate. But it has already changed what environmental scientists know about the chemicals around them, and it is the only approach capable of finding the next compound that nobody thought to measure. Doing it well means holding two attitudes at once: ambition about what can be discovered, and rigour about what has actually been shown. Chapter 1: Looking for What Nobody Listed Every analytical measurement begins with a decision about what to measure. In most environmental and product-safety chemistry that decision is made before the sample is collected, and it is made by a regulation, a contract or a method standard. The result is a system that is precise, auditable and blind to everything outside its list. Non-targeted screening does not abolish that decision. It postpones it, and it changes its character from a fixed choice to a sequence of increasingly specific questions asked of stored data. Understanding the three broad modes of screening, and what each can and cannot deliver, is the foundation for everything else. Three ways of asking a sample what it contains Targeted analysis starts with a list of compounds and a reference standard for each one. The instrument is typically a triple-quadrupole mass spectrometer operated in multiple reaction monitoring mode: the first quadrupole selects the precursor ion of a known compound, a collision cell fragments it, and the third quadrupole passes one or two characteristic fragments. Because the instrument ignores almost everything else, sensitivity is excellent and quantification is straightforward, using calibration curves and isotope-labelled internal standards. A targeted method answers the question "how much of compound X is present?" with high confidence. It cannot answer "what else is here?", and it cannot be reinterrogated later for compounds that were not programmed in advance, because the data for them were never recorded. Suspect screening starts with a list of compounds that might plausibly be present but for which the laboratory usually has no reference standard. The list may hold a few dozen pharmaceutical transformation products or tens of thousands of registered industrial chemicals. The instrument is a high-resolution mass spectrometer recording full-scan spectra, so the data contain every ionisable compound within the mass range. The analyst calculates the expected exact mass of each suspect's likely ions and searches the data for peaks at those masses, then tests each hit against further evidence: isotope pattern, fragmentation, retention time plausibility. The question is "which of these candidates can I find evidence for?" Non-targeted screening, in the strict sense, starts with no list at all. The analyst processes the full-scan data into features, each a combination of accurate mass, retention time and intensity, and then decides which features deserve attention, perhaps because they are intense, because they appear downstream of a discharge but not upstream, because they rise over time, because they carry a halogen isotope signature, or because they fall on a homologous series. Only then does identification begin, and it begins from the data rather than from a name. The question is "what is this signal, and does it matter?" In practice the boundary between suspect and non-targeted work is porous. A strictly non-targeted workflow still ends by searching candidate structures in databases, which is a form of suspect screening against an enormous list. A suspect screen often uncovers unexpected features that need non-targeted reasoning. Many authors use "non-target screening" or "non-targeted analysis" as an umbrella for both, and the US community has adopted the abbreviation NTA for the whole field. This book follows that practice, and distinguishes the modes where the distinction matters. The three modes differ in what they demand and what they return, and Table 1 sets these differences side by side. Table 1. How targeted, suspect and non-targeted screening differ. Aspect Targeted Suspect screening Non-targeted screening Starting point Fixed analyte list List of plausible candidates Data features, no list Reference standards Required for every analyte Needed only for final confirmation Needed only for final confirmation Typical instrument Triple quadrupole High-resolution full scan High-resolution full scan with MS/MS Main output Concentrations with uncertainty Candidate identities with confidence levels Prioritised unknowns with confidence levels Retrospective use Not possible Possible on stored data Possible on stored data Main risk Missing unlisted compounds False positives from mass coincidence Effort spent on irrelevant features The last column of that table deserves emphasis. The characteristic failure of targeted analysis is the false negative: a harmful compound that was present but not on the list. The characteristic failure of suspect screening is the false positive: a name attached to a peak because the masses agreed, when the peak was really something else of the same formula or something of a different formula that happened to fall within the mass tolerance. The characteristic failure of non-targeted screening is waste: thousands of features, most of them uninteresting, consuming analyst time while the important ones sit unexamined. Good practice in each mode is largely a matter of guarding against its own characteristic failure. Why environmental water and consumer goods are hard Two sample types dominate this book because they combine high stakes with extreme chemical complexity. Environmental water Surface water downstream of a city carries the combined output of domestic wastewater, industrial discharges, road runoff, agricultural drainage and atmospheric deposition. A single litre can contain pharmaceuticals and their human metabolites, personal-care ingredients, artificial sweeteners, corrosion inhibitors such as benzotriazoles, pesticides and their degradation products, flame retardants, plasticisers, surfactants in long homologous series, per- and polyfluoroalkyl substances, tyre-derived chemicals, and dissolved natural organic matter at concentrations orders of magnitude higher than all of the above combined. Many of these compounds are polar and persistent, which is exactly why they pass through wastewater treatment and why they are hard to extract. On top of the parent compounds sit their transformation products. Wastewater treatment oxidises, hydroxylates, demethylates and cleaves molecules. Ozonation and chlorination in drinking-water treatment produce their own by-products. Sunlight in the river adds photoproducts. A transformation product can be more mobile, more persistent or occasionally more toxic than its parent, and very few of them are on any regulatory list. For water utilities and environmental agencies, this is the practical argument for non-targeted methods: the compounds most likely to reach a drinking-water intake unnoticed are precisely those that nobody thought to target. The analytical challenges follow from the chemistry. Concentrations of individual micropollutants are typically in the nanogram-per-litre range, so samples must be concentrated, and every concentration step is selective. The matrix suppresses or enhances ionisation unpredictably. Natural organic matter contributes a dense background of ions. And the most mobile compounds, small, highly polar and often charged, are poorly retained on the reversed-phase columns that most laboratories use, so they elute early, crowded together with salts and matrix. Consumer goods Consumer goods present a different version of the same problem. A plastic food container, a child's toy, a textile or a cosmetic is a formulated material. It contains intentionally added substances, such as the polymer, plasticisers, antioxidants, light stabilisers, pigments, flame retardants and processing aids, and it contains non-intentionally added substances, known in food-contact regulation by the abbreviation NIAS. NIAS include impurities in raw materials, oligomers of the polymer, breakdown products of additives formed during processing, and reaction products between additives. European food-contact rules for plastics require that their risk be assessed, which in practice is impossible without some form of non-targeted analysis, because by definition nobody listed them. Groh and colleagues compiled a database of chemicals associated with plastic packaging in 2019 and counted well over four thousand substances, many with no hazard data. That count comes from literature and industry sources rather than from measurement, and it is certainly an underestimate of what actually migrates. The analytical task for a product-safety laboratory is therefore to find out what comes out of a material into food, saliva, sweat or indoor air, and to decide which of those substances need a proper risk assessment. The analytical challenges here include very high concentrations of a few dominant components that can saturate the detector and mask trace constituents, polymer oligomers that produce long ladders of related peaks, compounds that are volatile or poorly ionisable by electrospray and so require gas chromatography, and extraction choices that strongly bias what is seen. Migration testing into food simulants, such as ethanol-water mixtures or vegetable oil substitutes, reflects real exposure better than total extraction with an aggressive solvent, but it yields lower concentrations and a different chemical profile. What high-resolution mass spectrometry adds Low-resolution mass spectrometers report masses to roughly the nearest integer or tenth of a unit. That is enough to distinguish compounds when the chromatography and the list of targets do most of the work, but it cannot tell a formula from its many isobaric neighbours. High-resolution instruments, principally time-of-flight, Orbitrap and Fourier-transform ion cyclotron resonance analysers, report masses to a few thousandths of a unit or better. The practical consequence is that the measured mass constrains the elemental composition. A simple comparison shows why. Consider two ions that a low-resolution instrument would both call mass 299. The protonated molecule of 6PPD-quinone, the tyre-derived salmon toxicant, has the formula C18H23N2O2+ and a monoisotopic mass of 299.1754. The deprotonated molecule of perfluorobutanesulfonic acid, a short-chain PFAS, is C4F9O3S− at 298.9430, and its nominal mass is also 299. The difference between these two, about 0.23 units, is enormous by high-resolution standards. Much more demanding are pairs of formulas that differ by only a few thousandths of a unit, such as those related by swapping CH4 for O, a difference of 0.0364 units, or swapping C3 for SH4, a difference of 0.0034 units. A modern instrument measuring to within 2 parts per million at mass 300, which is about 0.0006 units, can distinguish the first pair easily and the second pair with care. The measured mass is only the beginning. The same full-scan spectrum records the isotope peaks, whose spacing and relative intensities reveal the number of carbon atoms and the presence of chlorine, bromine or sulfur. The mass defect, the small difference between the exact mass and the nearest integer, is characteristic of elemental composition: hydrogen-rich organic molecules have positive mass defects, while heavily fluorinated or brominated molecules have mass defects near zero or negative. And when the instrument also fragments ions, the product-ion spectra are recorded at high resolution too, so every fragment has its own formula. Crucially, full-scan high-resolution data can be stored and searched again later. A sample analysed in 2020 can be screened in 2026 for a compound first reported in 2025, provided the compound was ionised and retained under the original conditions. This retrospective capability, sometimes called digital archiving, turns each non-targeted run into a record of the sample's chemical composition as far as the method could see it. Chapter 9 returns to the idea. Where non-targeted screening earns its cost Non-targeted screening is expensive in instrument time, data processing and, above all, expert attention. It is not a replacement for targeted monitoring, which remains the right tool when the question is compliance with a limit for a known substance. Its value is highest in a small number of recurring situations. The first is discovery of unknown toxicants, often in combination with effect-directed analysis, where a biological assay is used to follow toxicity through successive fractionations of a sample until the active fraction is simple enough to identify its constituents. The 6PPD-quinone story is the landmark case, but the same approach has been used for endocrine-active compounds in wastewater, for mutagenic compounds in industrial effluent and for toxic drinking-water disinfection by-products. The second is early warning and trend detection. Water agencies on large rivers, notably the Rhine, run high-resolution screening at monitoring stations and look for new or rising features, which can reveal an accidental industrial release within days. The strength here is not identification of every feature but the ability to notice that something has changed and to prioritise it. The third is source tracking and fingerprinting. Mixtures of chemicals from a given source, whether a particular wastewater plant, a fire-training site with aqueous film-forming foam, or a specific product type, carry recognisable patterns of features. Homologous series revealed by Kendrick mass defect analysis are especially useful for fluorinated foams, surfactants and polymers. The fourth is exposure and product characterisation: finding out what migrates from food-contact materials, toys, textiles and building products, including the non-intentionally added substances that no formulation sheet will disclose. The fifth is prioritisation for regulation and targeted monitoring. Suspect screening across many sites can show which unregulated compounds are frequently detected, feeding lists of substances that deserve proper targeted methods and hazard assessment. The European NORMAN network of reference laboratories has built much of its work around exactly this loop, from screening to prioritisation to targeted follow-up. In each case the output of non-targeted screening is a decision about where to spend further effort, and that decision is only as good as the confidence attached to each identification. A tentative identification that sends a laboratory to buy a reference standard is useful even if it turns out to be wrong. A tentative identification that ends up in a risk assessment as if it were confirmed is dangerous. The rest of this book is about building identifications that can bear the weight placed on them. What no method can see The phrase "non-targeted" invites an assumption that the method sees everything. It does not, and a study that forgets this will misreport absence as evidence. Every stage of the workflow filters the chemical universe, and the filters compound. Sampling and extraction decide which compounds reach the instrument at all. A reversed-phase solid-phase extraction cartridge retains moderately polar and non-polar compounds well and loses many small, highly polar or permanently charged ones. Evaporation steps lose volatiles. Filtration removes particle-bound compounds. The chromatographic column then decides which compounds are separated and which co-elute in the void volume. The ionisation source decides which compounds become ions: electrospray favours molecules that carry or accept a charge readily in solution, such as amines, acids and sulfonates, and does very poorly for non-polar hydrocarbons, many chlorinated aromatics and most siloxanes. Gas chromatography with electron ionisation covers much of what electrospray misses, but only for compounds volatile and stable enough to pass through a heated column. Finally, the data-processing software decides which signals count as peaks, and an intensity threshold set to keep the feature list manageable will discard the trace compounds that may matter most. The practical consequence is that a non-targeted study defines a region of chemical space, bounded by extraction, separation, ionisation and processing, within which it can report detections with some confidence. Outside that region it can report nothing. Good studies characterise their region explicitly, often by spiking a mixture of reference compounds spanning a range of polarity, size and functional groups and reporting which were recovered. The US EPA's Non-Targeted Analysis Collaborative Trial, known as ENTACT, did this on a large scale, distributing mixtures of several hundred known substances to participating laboratories and comparing what each method found. The lesson from such exercises is consistent: different laboratories, using different but individually reasonable methods, see substantially different subsets of the same mixture. There is no universal method, and the absence of a compound from a non-targeted report means only that it was not found by that method at that sensitivity. This limitation is also an argument for combining techniques. A study of a consumer product that uses only liquid chromatography with positive-mode electrospray might miss most acidic additives and all volatile ones. Adding negative mode and a gas-chromatographic method with high-resolution detection roughly doubles the effort and considerably widens the window. Whether that is worthwhile depends on the question, which is why the question comes first in the workflow described below. The shape of a workflow Despite the diversity of applications, most non-targeted studies follow the same broad sequence, and it is worth fixing it in mind before the details. Define the question: what decision will the results inform, and which compounds would matter? Collect and prepare samples with blanks and quality-control samples designed in from the start. Acquire full-scan high-resolution data, usually with fragmentation, and usually in both positive and negative ionisation modes. Process the raw data into features: detect peaks, group isotopes and adducts, align across samples, remove blank-derived features. Prioritise features by intensity, occurrence pattern, trend, isotope signature, mass defect or suspect list membership. Assign molecular formulas using accurate mass and isotope evidence. Propose structures using fragmentation, databases, in silico tools and retention behaviour. Confirm the most important identifications with reference standards where possible. Report each identification with an explicit confidence level and enough metadata for others to judge it. Most of the intellectual difficulty lies in steps 5 to 8, and most of the practical errors originate in steps 2 to 4. A contaminated blank, a poorly chosen extraction or an aggressive peak-picking threshold can remove or invent compounds long before anyone looks at a spectrum. That is why the next two chapters deal with the instrument and the data pipeline before the book turns to identification itself. Chapter 2: What the Instrument Actually Measures High-resolution mass spectrometers are often described by two numbers on a specification sheet: a resolving power, such as 120,000, and a mass accuracy, such as "below 3 ppm". Both numbers are real, both matter, and they are routinely confused with each other. A laboratory that understands the difference, and understands how each degrades under real conditions, can judge its own data. A laboratory that does not will either trust results it should question or reject results that are perfectly sound. This chapter builds that understanding from the arithmetic up. Exact mass, nominal mass and mass defect The nominal mass of an ion is the sum of the integer masses of its most abundant isotopes: 12 for carbon, 1 for hydrogen, 14 for nitrogen, 16 for oxygen, 19 for fluorine, 32 for sulfur, 35 for chlorine. The monoisotopic exact mass uses the accurately known masses of those same isotopes. By definition carbon-12 weighs exactly 12. Hydrogen-1 weighs 1.007825, nitrogen-14 weighs 14.003074, oxygen-16 weighs 15.994915, fluorine-19 weighs 18.998403, sulfur-32 weighs 31.972071, chlorine-35 weighs 34.968853 and bromine-79 weighs 78.918337, all in unified atomic mass units, also called daltons. The difference between exact and nominal mass is the mass defect. Hydrogen has a positive mass defect of about 0.0078 per atom, so hydrogen-rich molecules sit above their nominal mass. Oxygen, fluorine, sulfur, chlorine and bromine have negative mass defects, so molecules rich in these elements sit below, or only slightly above, their nominal mass. A long-chain hydrocarbon of nominal mass 400 will have an exact mass near 400.4 or 400.5; perfluorooctanoic acid, with fifteen fluorine atoms, has a neutral exact mass of 413.9737, a mass defect of about −0.03. That simple observation is the seed of the Kendrick analysis developed in Chapter 6, and it already suggests a way to spot halogenated compounds in a crowded spectrum. When calculating the mass of an ion rather than a neutral molecule, the mass of the electron, 0.000549, must be accounted for. A protonated molecule, written [M+H]+, weighs the neutral mass plus a hydrogen atom minus an electron, which is the neutral mass plus 1.007276. A deprotonated molecule, [M−H]−, weighs the neutral mass minus 1.007276. Ignoring the electron introduces an error of about 0.5 millidaltons, which at mass 200 is almost 3 parts per million: large enough to push a correct assignment outside a tight tolerance window. Most software handles this automatically, but anyone checking an identification by hand should be aware of it. Electrospray ionisation frequently produces ions other than the simple protonated or deprotonated molecule. Sodium adducts [M+Na]+ add 22.989218 to the neutral mass; ammonium adducts [M+NH4]+ add 18.033823; in negative mode, chloride adducts [M+Cl]− add 34.969402 and formate adducts [M+HCOO]− add 44.998201. Molecules may also lose water in the source, appear as dimers such as [2M+H]+, or carry two charges. A single compound can therefore generate several features at different masses but the same retention time. Recognising these as one compound, a process called componentisation, is essential both for avoiding double counting and for deducing the neutral mass of an unknown: if two co-eluting peaks differ by exactly 21.981942, the mass difference between Na and H, they are very likely the sodium and proton adducts of the same molecule. Mass accuracy and the arithmetic of parts per million Mass accuracy is the closeness of a measured mass to the true mass. It is almost always expressed as a relative error in parts per million: error (ppm) = (measured - theoretical) / theoretical x 1,000,000 Take carbamazepine, an anticonvulsant drug that is one of the most persistent pharmaceutical markers of wastewater in surface waters. Its formula is C15H12N2O, and its protonated molecule has a theoretical monoisotopic mass of 237.10224. Suppose the instrument reports 237.10261. The difference is 0.00037, and dividing by 237.10224 and multiplying by a million gives an error of +1.56 ppm. Suppose instead it reports 237.10150: the error is −3.12 ppm. The same absolute error means different relative errors at different masses. An absolute error of 1 millidalton is 10 ppm at mass 100, 2 ppm at mass 500 and 1 ppm at mass 1000. For small molecules, some practitioners therefore state tolerances as the larger of a ppm value and an absolute value, for instance "5 ppm or 1 mDa", to avoid unrealistically narrow windows at low mass. A tolerance window is a statement about how many candidate formulas are admitted. At mass 237.10224 a ±5 ppm window spans roughly ±0.0012, or 2.4 millidaltons in total. Within that window, allowing only carbon, hydrogen, nitrogen and oxygen, a small number of chemically sensible formulas fit; allowing sulfur, phosphorus, chlorine and fluorine as well, more do. As the mass increases, the number of possible formulas within any fixed ppm window grows rapidly, because there are more ways to combine atoms. This was the point made forcefully by Kind and Fiehn in 2006 and 2007: at masses above about 500, even sub-ppm accuracy does not by itself give a unique formula, and isotope information becomes indispensable. Chapter 5 takes up that argument. Mass accuracy on a real sample depends on more than the instrument's specification. It depends on calibration and on how recently the instrument was calibrated. Many methods use a lock mass, a known background ion or a deliberately infused reference compound, to correct drift scan by scan. It depends on ion statistics: a peak built from a few hundred ions has a less precise centroid than one built from a hundred thousand, so trace compounds typically show larger errors than abundant ones. It depends on space charge in trapping instruments, where too many ions in the trap shift frequencies and hence masses. It depends on detector saturation in time-of-flight instruments, where intense peaks can be distorted. And, most insidiously, it depends on resolution: if two ions of slightly different mass are not resolved, the instrument reports a single peak whose centroid lies somewhere between them, and that apparent mass matches neither. A measured error of 6 ppm on an otherwise well-calibrated instrument is often a sign of an unresolved interference rather than of a bad calibration. The size of the effect is easy to estimate. Suppose a compound of interest gives an ion at 300.1000 and a matrix ion of equal intensity sits at 300.1030, three millidaltons higher. At a resolving power of 60,000, each peak is about 0.005 wide at half height, wider than the gap between them, so the instrument sees one broadened peak with its centroid near 300.1015. Measured against the true mass of the compound of interest, that is an error of +5 ppm. If the interfering ion is only a tenth as intense, the centroid shifts less, perhaps by half a ppm, and the error is hidden inside normal scatter. Raising the resolving power to 240,000 narrows each peak to about 0.00125 and separates the two cleanly, restoring the correct mass. This is why a peak whose shape is broader than expected for its mass, or whose apparent mass drifts across its chromatographic profile, deserves a second look. The practical rule is that a laboratory should know its own mass-error distribution. Spiking isotope-labelled internal standards into every sample and recording their mass errors across the run gives a direct, sample-specific estimate. If internal standards are consistently within ±2 ppm, a candidate at +4.5 ppm should be treated with suspicion even though it falls inside a nominal 5 ppm window. Resolving power and why it is a different thing Resolving power describes the instrument's ability to separate two ions of similar mass. The most common definition is the full-width-at-half-maximum definition: resolving power equals the mass of a peak divided by its width at half its height. R = m / delta_m (FWHM) A peak at mass 300 with a width of 0.005 at half height has a resolving power of 60,000. Two peaks of equal height separated by roughly one peak width at half maximum will appear as a partly merged pair; to see them cleanly separated, especially when one is much smaller than the other, requires a larger separation, commonly two or more times the half-height width. Resolving power and mass accuracy are related but not interchangeable. A narrow peak makes it easier to locate the centroid precisely, and resolving power prevents the interference problem described above, so higher resolving power generally supports better accuracy on real samples. But a well-calibrated instrument can measure an isolated, intense, fully resolved peak to within 1 ppm at modest resolving power, and a high-resolution instrument that has drifted can be both highly resolving and several ppm wrong. The two specifications answer different questions: resolving power asks "can I tell these two ions apart?", mass accuracy asks "how close is the number to the truth?" The resolving power needed for a given task can be calculated directly. To separate two ions whose masses differ by Δm at mass m, the instrument needs a resolving power of at least m/Δm, and in practice about double that for peaks of unequal intensity. Consider the ion at mass 300 that contains one carbon-13 atom, and a different ion of the same nominal mass that contains a nitrogen-15 atom instead. The mass shift for carbon-13 over carbon-12 is 1.003355; for nitrogen-15 over nitrogen-14 it is 0.997035. The difference is 0.00632. Dividing 300 by 0.00632 gives a required resolving power of about 47,500 to begin separating them, and closer to 100,000 for comfortable separation of a small nitrogen-15 peak beside a large carbon-13 peak. That calculation, and others like it, underlie the isotope fine structure methods of Chapter 5. The three families of high-resolution analyser Three kinds of mass analyser dominate high-resolution screening, and they differ in how resolving power behaves across the mass range and in speed, cost and practicality. Table 2 summarises the contrasts qualitatively; exact figures vary by model and generation. Table 2. The main high-resolution mass analysers compared. Property Time-of-flight (Q-TOF) Orbitrap FT-ICR Measuring principle Flight time over fixed path Frequency of axial oscillation Cyclotron frequency in magnetic field Resolving power versus m/z Roughly constant Falls with square root of m/z Falls in proportion to m/z Scan speed Very fast Slows as resolution rises Slow at highest resolution Isotope fine structure Limited Possible at highest settings Routine Typical role in screening Fast LC and GC screening Workhorse of LC screening Specialist, complex mixtures Time-of-flight instruments accelerate ions with a fixed energy and time their arrival at a detector after a flight path, usually folded by one or more ion mirrors called reflectrons. Lighter ions arrive first. Modern quadrupole time-of-flight instruments offer resolving powers in the tens of thousands, roughly constant across the useful mass range, and they acquire spectra very quickly, which suits fast chromatography and data-independent fragmentation schemes. Their mass accuracy depends heavily on frequent calibration, typically with a continuously infused reference compound. Orbitrap analysers trap ions in an electrostatic field around a spindle-shaped central electrode, where they oscillate along the axis with a frequency that depends on their mass-to-charge ratio. The image current induced by the oscillating ions is recorded as a transient signal and converted into a spectrum by Fourier transformation. Resolving power increases with the length of the recorded transient, so higher resolution means slower scans. It also falls with the square root of m/z, which is why Orbitrap resolving power is always quoted at a reference mass, commonly m/z 200. An instrument set to 120,000 at m/z 200 delivers about 60,000 at m/z 800. The highest settings on recent instruments are high enough to resolve some isotope fine structure for small molecules, at the cost of scan speed. Fourier-transform ion cyclotron resonance analysers trap ions in a strong magnetic field, where they orbit at a cyclotron frequency inversely proportional to m/z. They offer the highest resolving power of any analyser, enough to resolve fine structure routinely and to separate thousands of components in natural organic matter or petroleum by direct infusion without chromatography. The cost, size and the need for superconducting magnets restrict them to specialist laboratories, and their slower acquisition makes them less convenient for coupling to fast chromatography. Much of what is known about the molecular composition of dissolved organic matter, and much of the development of Kendrick mass defect analysis for complex mixtures, comes from FT-ICR work. Ionisation and separation: the front end The analyser only measures what reaches it. For liquid chromatography, the dominant ion source is electrospray ionisation, which produces ions from charged droplets and works best for polar compounds that are already charged or easily protonated or deprotonated in solution. Screening methods usually acquire in both positive mode, favouring bases such as amines and many pharmaceuticals, and negative mode, favouring acids, phenols, sulfonates and most PFAS. Atmospheric pressure chemical ionisation and atmospheric pressure photoionisation extend coverage to less polar compounds, such as some flame retardants and polycyclic aromatic compounds. For volatile and semi-volatile compounds, gas chromatography coupled to high-resolution mass spectrometry with electron ionisation is the complement. Electron ionisation produces extensive, reproducible fragmentation, which is why large nominal-mass spectral libraries exist for it, but it often destroys the molecular ion. Softer techniques, such as chemical ionisation or atmospheric pressure chemical ionisation interfaced to gas chromatography, preserve the molecular ion and so give the accurate molecular mass needed for formula assignment. Consumer-product screening, where many additives, residual monomers and fragrance compounds are volatile, relies heavily on GC-based methods. Chromatography contributes more than separation. Retention time is an independent physical property that can support or refute an identification, particularly when predicted retention is available, as Chapter 7 describes. Ion mobility spectrometry, now integrated into several commercial high-resolution platforms, adds a further separation in the gas phase by shape and size and yields a collision cross section for each ion, another measurable property that can be compared with library or predicted values. Fragmentation: data-dependent and data-independent acquisition Identification beyond formula level requires tandem mass spectrometry: isolating a precursor ion, fragmenting it by collision with a neutral gas, and recording the product ions. Two acquisition strategies dominate. In data-dependent acquisition, the instrument records a full-scan spectrum, picks the most intense ions that meet certain criteria, isolates each one in turn and records its fragment spectrum. The resulting spectra are clean, because each is linked to one isolated precursor, but coverage is limited: in a complex sample, trace compounds may never be selected. Inclusion lists that prioritise suspect masses and exclusion lists that suppress known background ions help, and repeated injections with dynamic exclusion can increase coverage. In data-independent acquisition, the instrument fragments everything, either all ions at once or ions within successive wide isolation windows, and records composite fragment spectra. Every compound is fragmented, but the fragment spectra are mixtures, and fragments must be associated with their precursors computationally by matching the shapes of their chromatographic peaks. Software such as MS-DIAL, published by Tsugawa and colleagues in 2015, was developed specifically to deconvolute such data. Neither strategy is universally superior. Data-dependent acquisition gives more trustworthy spectra for the compounds it selects; data-independent acquisition guarantees that some fragment information exists for every compound, which matters for retrospective screening of archived data. Many laboratories now run both, or use hybrid schemes that trigger targeted fragmentation for suspects while acquiring broad windows for everything else. Whatever the scheme, the collision energy matters. Fragment spectra change markedly with energy, and a library spectrum recorded at one energy may match a sample spectrum recorded at another only poorly. Stepped or ramped collision energies, which combine spectra across a range, produce richer spectra that are more likely to contain both the fragile and the robust fragments of a molecule. When comparing a sample spectrum with a library or with a reference standard, like should be compared with like. The instrument, then, delivers four kinds of evidence for each compound it detects: an accurate mass, an isotope pattern, a retention time and, if selected, a fragment spectrum. The next chapter follows those signals through the data pipeline that turns raw files into a list of features, where many identifications are quietly won or lost before any chemist looks at a spectrum. Chapter 3: From Sample to Feature List Identification gets most of the attention in non-targeted screening because it is where the intellectual puzzles lie. But the compounds a study can identify are fixed long before identification starts: by where and how samples were collected, how they were extracted, what went into the blanks, and how the raw data were turned into a list of features. Errors at these stages are silent. A contaminant introduced by a sampling bottle looks exactly like a pollutant. A compound lost during evaporation leaves no gap in the data to show that it was ever there. A peak-picking threshold set too high removes trace signals without comment. This chapter follows the path from sample to feature list and identifies the decisions that most affect what comes out at the end. Designing the study before collecting anything The first design decision is the comparison. Non-targeted data are most informative when they contrast samples: upstream against downstream of a discharge, influent against effluent of a treatment plant, a product before and after ageing, a food simulant exposed to packaging against an unexposed simulant, a time series at a monitoring station. A single sample analysed in isolation produces thousands of features with no way of deciding which matter. A well-chosen contrast turns the same features into a ranked list: those present only downstream, those increasing through treatment, those that migrate. The second decision is replication. Features that appear in one of three replicate injections are usually noise or carry-over; features that appear in all three with consistent intensity are real signals in the extract. Replicate extractions, not just replicate injections, are needed to separate analytical variability from preparation variability. For comparisons between sites or treatments, replicate field samples are needed to separate environmental variability from both. The third decision is quality control, which in non-targeted work has three main components. Blanks reveal contamination. A field blank, prepared from clean water carried to the site and handled like a sample, captures contamination from containers and handling. A procedural blank, taken through the whole extraction, captures contamination from solvents, cartridges and glassware. An instrument blank, a solvent injection, captures carry-over and background from the chromatographic system itself. Internal standards, usually isotope-labelled analogues of compounds spanning a range of polarity and retention, are spiked into every sample to monitor retention time stability, mass accuracy and signal intensity across the batch. Pooled quality-control samples, made by mixing aliquots of all samples, are injected at regular intervals to measure drift and to provide a representative sample for acquiring fragmentation data. Injection order matters too. Samples run in a fixed order, all upstream samples first and then all downstream samples, confound location with instrument drift. Randomising the order, or blocking it so that each comparison is balanced across the run, protects against spurious differences. Sampling and extraction for water For surface water and wastewater, grab samples capture a moment and composite samples, collected over 24 hours or longer by automatic samplers, capture an average. Wastewater concentrations vary strongly with the daily rhythm of domestic activity, so composites are usually preferable for effluents. Passive samplers, devices deployed in the water for days or weeks that accumulate compounds onto a sorbent, integrate over longer periods and can reach lower detection limits, though converting their accumulated amounts to water concentrations requires calibration data that most non-targeted compounds lack. Containers must be chosen with contamination in mind. Many plastic containers leach plasticisers, antioxidants and slip agents that are exactly the kinds of compounds a screening study might report. Fluoropolymer parts, common in sampling equipment and chromatographic systems, can release fluorinated compounds, which is why PFAS analysis often replaces them with polyethylene or polyether ether ketone components and installs a delay column between the solvent pumps and the injector so that system-derived PFAS elute later than those from the sample. Extraction is the stage that most strongly shapes the chemical window. The most widely used approach is solid-phase extraction on a hydrophilic-lipophilic balanced polymeric sorbent, which retains a broad range of moderately polar to non-polar compounds. Its weakness is very polar and ionic compounds, which pass through. Laboratories seeking wider coverage use multilayer cartridges that stack a balanced sorbent with mixed-mode cation and anion exchange materials and sometimes graphitised carbon, each capturing a different part of the polarity range. The penalty is more matrix and more complex elution. The alternative is to avoid extraction altogether. Direct injection of filtered water, often in volumes of hundreds of microlitres, preserves the most polar compounds and avoids selective losses, at the cost of sensitivity and of loading salts and matrix onto the column. For relatively clean waters, such as groundwater or drinking water, direct injection combined with a sensitive instrument can reach the concentrations of interest. Freeze-drying or evaporative concentration retains polar compounds but also concentrates salts, which cause severe ion suppression. Whichever route is chosen, the extraction should be characterised by recovery experiments with a mixture of reference compounds spanning the relevant polarity range. Those recoveries define the chemical window of Chapter 1 in concrete terms and should be reported. Sampling and extraction for consumer goods Consumer products raise a prior question: extract what, into what? A total extraction, in which the material is ground and exhaustively extracted with an organic solvent such as dichloromethane, acetonitrile or a mixture, reveals as much as possible of the material's composition, including additives that would never migrate in use. A migration test exposes the material to a food simulant, artificial saliva or artificial sweat under defined time and temperature conditions and analyses what comes out. For food-contact materials, the European regulation on plastics specifies simulants for different food types, from 10% ethanol for aqueous foods to 95% ethanol or isooctane as substitutes for fatty foods. Migration tests are closer to real exposure and are what a risk assessment ultimately needs; total extractions are useful for building a suspect list of what could migrate. Polymer materials raise specific issues. Extracts contain oligomers of the polymer itself, cyclic and linear, often in long series that dominate the feature list. Polyesters yield cyclic oligomers of their repeating units; polyamides yield cyclic lactam oligomers; polyurethanes and polyethers yield series differing by their repeating units. These are genuine migrants and some are of toxicological interest, but they can overwhelm data processing. Kendrick mass defect analysis, described in Chapter 6, is the natural tool for recognising and grouping them. High-concentration components, such as a plasticiser present at percent levels, can saturate the detector and spread through the chromatogram by carry-over. Dilution series, injecting the same extract at several dilutions, help: abundant compounds are measured accurately in the dilute injection and trace compounds are detected in the concentrated one. Volatile and semi-volatile migrants, including residual solvents, monomers, fragrance ingredients and breakdown products of antioxidants, are often better suited to gas chromatography. Headspace or solid-phase microextraction sampling of the product, followed by GC with high-resolution detection, captures what the product releases into air, which is directly relevant for indoor exposure from furniture, building materials and toys. Turning raw data into features A modern high-resolution run produces a raw file containing thousands of spectra, each with tens of thousands of data points. The first processing steps convert these into a feature table: a matrix with one row per feature, characterised by m/z and retention time, and one column per sample, holding intensities. Raw files are vendor-specific. Converting them to the open mzML format, commonly with the msconvert tool of the ProteoWizard project, allows processing by open software and supports sharing and archiving. Most processing uses centroided spectra, in which each peak is reduced to a single m/z and intensity, which reduces file size and simplifies downstream steps; the profile data should be kept, however, because centroiding algorithms can merge or split closely spaced peaks, and profile data are needed to check isotope fine structure. Peak picking finds chromatographic peaks in extracted ion chromatograms. Algorithms differ in how they build ion traces, how they model peak shapes and how they separate noise from signal. The most widely used open tools include XCMS, published by Smith and colleagues in 2006, MZmine, now in its third major version described by Schmid and colleagues in 2023, and MS-DIAL. Environmental laboratories also use workflow packages such as patRoon, described by Helmus and colleagues in 2021, which wraps several algorithms in a common framework for environmental screening, as well as a range of commercial software supplied by instrument vendors. Parameter choices, in particular the minimum peak intensity, the mass tolerance for building ion traces and the expected peak width, determine how many features are found. Too permissive, and the table fills with noise; too strict, and trace compounds disappear. There is no universally right setting, and reporting the settings used is essential. Alignment matches features across samples, correcting for small retention time shifts and mass drifts so that the same compound in different samples ends up in the same row. Gap filling then revisits samples where a feature was not detected and integrates the signal at the expected position, distinguishing true absence from a peak that fell just below the picking threshold. Componentisation groups features that belong to the same compound: isotope peaks, adducts, in-source fragments and multiply charged ions. After componentisation, one compound should be one row, with its related ions annotated. Without it, a single abundant compound can contribute a dozen features and dominate any statistical analysis, and the analyst may try to identify a sodium adduct or an in-source fragment as if it were a separate molecule. Blank subtraction removes features that are present in blanks at comparable intensity. A common approach retains a feature only if its intensity in samples exceeds its intensity in blanks by a chosen factor, often somewhere between three and ten. The choice matters: a strict factor removes contaminants effectively but also removes genuine compounds that happen to have a small background, such as common plasticisers that are both pollutants and laboratory contaminants. Replicate and occurrence filters remove features that are not reproducible across replicates or that occur in too few samples to be interpretable. A typical environmental water sample processed this way yields thousands to tens of thousands of features across both ionisation modes, even after filtering. No laboratory can identify them all. Prioritisation is therefore not an optional refinement but the step that decides which part of the data will ever be examined. Prioritisation: choosing what to identify Prioritisation strategies fall into a few families, and most studies combine several. Intensity-based prioritisation starts with the most abundant features. It is simple and makes identification easier, because intense peaks give good spectra, but abundance in the mass spectrometer is not abundance in the sample, since ionisation efficiencies differ by orders of magnitude between compounds, and neither is it toxicity. Statistical and pattern-based prioritisation uses the study design. Features that increase significantly downstream of a discharge, that persist through treatment, that appear after a known event, or that correlate with a biological effect across samples are promoted. Time-series analysis at monitoring stations flags features that appear suddenly or trend upward. Hierarchical clustering can group features with similar spatial or temporal patterns, which often share a source. Chemistry-based prioritisation uses signatures in the mass spectrum itself. Features whose isotope patterns indicate chlorine or bromine, features with negative mass defects characteristic of heavy fluorination, and features that sit on homologous series in a Kendrick analysis are promoted because halogenated and persistent compounds are disproportionately of concern. Specific fragment ions or neutral losses, recorded in data-independent acquisition, can flag compound classes: sulfate and glucuronide conjugates, organophosphates, or perfluoroalkyl chains. Suspect-based prioritisation promotes features that match entries on a suspect list, which is the subject of the next chapter. Hazard-based prioritisation is the most ambitious: estimating toxicity for features whose identity is still uncertain, using predictions from tentative structures or from spectral features directly. Such approaches are active research areas and are best treated as supplements to, rather than replacements for, the strategies above. The choice of prioritisation strategy is a choice about which kind of compound the study is most likely to find. An intensity-driven study will find abundant, easily ionised compounds; a halogen-driven study will find halogenated ones. Neither is wrong, but the report should say which was used, so that readers understand why certain compounds appear and others do not. A worked pass through prioritisation Suppose a laboratory screens triplicate composite samples from a river 1 km upstream and 1 km downstream of a municipal treatment plant, together with the plant effluent itself, a field blank and a procedural blank, in negative-mode electrospray. After peak picking, alignment and componentisation, the software reports several thousand compound-level features. The analyst then applies a sequence of filters, each one stated in advance. First, blank filtering: any feature whose mean intensity in samples is less than five times its mean intensity in the procedural blank is removed. This eliminates, among other things, a family of fatty acids and a sulfonate surfactant that turn out to come from the cartridges and the laboratory detergent. Second, replicate filtering: features must be present in all three replicates of at least one sample type. Third, the contrast: features must be at least five times more intense downstream than upstream and must also be present in the effluent. What remains is a list of a few hundred features plausibly contributed by the plant. Now chemistry-based filters sharpen the list further. The analyst sorts the plant-derived features by the ratio of their A+2 isotope peak to their monoisotopic peak and flags those with ratios near 32 per cent, the signature of one chlorine atom, or near 64 per cent, suggesting two. Separately, features with mass defects below zero at masses above 250 are flagged as candidates for heavy fluorination. Finally, the list is cross-checked against a suspect list of pharmaceuticals and their transformation products. The result is not a set of identifications but a short, reasoned queue: perhaps a few dozen features, each with a stated reason for being there. One of them is a feature at m/z 294.0094 with a two-chlorine isotope pattern that matches the deprotonated molecule of diclofenac, a common anti-inflammatory drug, within 1 ppm; another is a chlorinated feature 15.9949 higher, the mass of one oxygen atom, suggesting a hydroxylated transformation product. The first is a straightforward suspect hit; the second is a genuine non-targeted lead. The value of the pipeline lies in having brought both to the analyst's attention for defensible reasons, and in having documented every filter so that another laboratory can see what might have been excluded along the way. Keeping the pipeline honest Three practices make the whole pipeline more trustworthy. The first is spiking known compounds into a subset of samples at environmentally relevant levels and tracing them through processing. If a spiked compound fails to appear as a feature, the parameters are too strict or the extraction loses it; if it appears as several features, componentisation is failing. The second is inspecting raw data for every feature that reaches the identification stage: looking at the extracted ion chromatogram, the peak shape, the isotope pattern and the presence in blanks. A surprising number of prioritised features turn out to be artefacts when examined directly. The third is versioning and reporting every processing parameter and software version, so that results can be reproduced and so that later reprocessing, for retrospective screening, starts from a known state. At the end of this pipeline, the analyst holds a feature table and a short list of prioritised features, each with an accurate mass, a retention time, an isotope pattern and, ideally, a fragment spectrum. The work of turning those features into chemical identities begins with the most tractable case: checking them against lists of compounds that are already suspected to be present. Hashtags: #NonTargetedChemicalScreening #HighResolutionMassSpectrometry #EnvironmentalToxins #NonTargetedAnalysis #SuspectScreening #EnvironmentalChemistry #AccurateMass #MassAccuracy #ResolvingPower #IsotopePatterns #IsotopeFineStructure #KendrickMassDefect #TandemMassSpectrometry #StructureElucidation #SchymanskiConfidenceLevels #NORMANSuspectList #CompToxChemicalsDashboard #FeaturePrioritization #ChemicalIdentification #6PPDQuinone #PFASScreening #TransformationProducts #RetrospectiveScreening #ConsumerProductScreening #FutureOfNonTargetedScreening