American Dynasty, a commercial trawler, departed Seattle one day in May to fish for hake off the Washington coast. Before leaving, its 122 crew members were screened for the coronavirus using the highly accurate polymerase chain reaction (P.C.R.) method, and all the results came back negative. But because those tests are “good but not perfect,” in the words of Jesse Bloom, a professor at the Fred Hutchinson Cancer Research Institute in Seattle, they missed at least one case: Somehow SARS-CoV-2 found its way on board.
When a crew member fell seriously ill, the vessel returned to port, and almost everyone was tested for the virus again. The before-and-after results for 120 of the crew members were made available to Bloom and colleagues, who published a study about them in The Journal of Clinical Microbiology in August. In addition to the P.C.R. tests, the pre-voyage screenings also looked for neutralizing antibodies, or proteins generated by the immune system after exposure to the virus, which suggest that a person has been infected previously. Three crew members, it turned out, had those antibodies at the start of the trip. Of the 117 crew members who did not, 103 tested positive for the virus when they got back to shore — an 88 percent infection rate. If you were to randomly select three names from the ship’s manifest, the odds that all three would have tested negative are about 0.2 percent. Yet all three sailors with antibodies were spared.
The finding is believed to be the first direct evidence that antibodies protect against SARS-CoV-2 infection in humans, and it offers clues about what sort of concentrations might be needed to confer immunity. The amounts of antibodies present in the three sailors are “pretty attainable by the vaccines” in development, says Alex Greninger, a virologist at the University of Washington School of Medicine and the study’s senior author. He says that data makes him “more optimistic” that these vaccines might work. In a commentary on the study, Danny Altmann, a professor of immunology at Imperial College London, called it “a remarkable, real-life, human experiment.”
When researchers examine past events in order to understand why some things happen to certain people and not others, they are conducting “observational studies.” These are often the only way to assess what behavioral and policy measures prevent the spread of the coronavirus. Unfortunately, they can’t prove cause and effect. Unlike randomized controlled trials, in which subjects are assigned by chance to “experimental” and “control” groups, ensuring the only difference between them is the intervention under investigation, observational studies always include the possibility that other factors are influencing the outcomes. Greninger points out that he can’t prove that everyone on American Dynasty was exposed to the virus; perhaps the three with antibodies, along with their 14 uninfected colleagues, never encountered it. Or maybe, says Anupam B. Jena, a professor of health care and policy at Harvard Medical School, they were younger and less susceptible to infection. (The researchers weren’t privy to information about the crew members beyond their samples.) To prove that the antibodies protected them, Jena says, “you have to show that the people who had the antibodies were similar in every other way to the people who didn’t have antibodies.”
The threat posed by the virus, though, has made randomized trials extremely difficult to do. While researchers have randomly assigned macaque monkeys to receive either a vaccine or a placebo and then exposed them to the virus (those with vaccine-produced antibodies were less likely to become infected), it would be unethical to perform this experiment on humans, given the risk of severe illness and death. Instead, vaccine trials are enrolling thousands of people, randomly giving them a vaccine or a placebo and waiting months until, in the normal course of their lives, a small percentage of them become infected; at that point, they’ll learn which group they were in. This process is much less efficient and provides less detailed information about the circumstances and biology of each infection than an experiment in which participants are closely monitored in a controlled setting.
That makes cases like the Dynasty’s, where a high percentage of people under fairly uniform conditions were infected with the virus while others weren’t, potentially very useful. Testing during an outbreak that infected some 700 people on the Diamond Princess cruise ship in February helped reveal that many infections never produced noticeable symptoms. “We’re trying to learn with limited evidence,” says Emily Oster, an economist at Brown University. “Digging into, What are the particular details of an incident that we can learn from? I think there’s a lot of value there.”
Such digging can help generate hypotheses. The classic example is the study by John Snow, an English physician, of a cholera outbreak in London in 1854. Plotting cases of the disease on a map and interviewing residents — “It’s like the original contact tracing,” Oster says — Snow began to suspect that a pump, where many of those getting sick got their water, was the culprit. But Snow had also tested his theory, that cholera was waterborne, by getting maps of households that were (randomly) served by one of two rival water companies, with different water sources, and noting which of them experienced cholera deaths. When the death rate for one company proved much higher than the other, it was clear that its water was the cause. (Sewage contamination was to blame.)
Such “natural experiments,” in which some event or factor has randomized participants into experimental and control groups, have been especially hard to find during the pandemic. The urgent need to stop the spread of the virus has led policymakers to change many variables at once — school and business closures and reopenings, mask ordinances. This makes it difficult to separate their effects. To find out if closing schools lowered community infection rates early in the outbreak, for example, you might look at demographically similar areas where schools closed either in mid-March or early April and compare their infection rates in early May. “But the places that have been hit the hardest might be the ones that are pulling the trigger earlier,” says Joseph Doyle, an economist at the M.I.T. Sloan School of Management — which can make it seem that closing schools leads to high infection rates, when actually an anticipated rise in infections caused the schools to close. Randomizing them would mean finding ones that closed for reasons “unrelated to anything about the health of the community,” Doyle says. For instance, several schools in Tennessee were hit by tornadoes in March and shuttered early, while neighboring ones stayed open. Comparing their community infection rates weeks later could approximate a randomized trial — if the storms didn’t meaningfully affect other local interactions.
Ideal randomization is rarely possible. “Sometimes if you need answers, you go with natural experiments that aren’t potentially as super clean, but we can learn from them,” Doyle says. “And then you back them up with theory.” On American Dynasty, for instance, antibody distribution wasn’t governed by chance weather. But there is previous evidence to support the theory that antibodies protect against infection — making it more plausible that they, as opposed to some other shared factor, protected the three sailors.
In a sense, all of us have been generating pandemic data for six months: following the policies enacted by local officials or not, sending our children to school or not. Most positive cases are not being rigorously investigated, which means we have a poor understanding of how often (for example) certain behaviors result in infection. Natural experiments can’t fix that problem. But looking for them could let us collect data or use what we do have to arrive more efficiently at answers to questions as they arise: To what extent do steps like requiring masks prevent the virus from spreading in schools? Does a new drug improve Covid-19 outcomes? What kinds of messages motivate people to get vaccinated? What vaccine works best and for whom? “These natural experiments — they’re there,” Jena says.
