Why do so many astronomy discoveries fail to live up to the hype?

"Breakthroughs" come along regularly but often don’t hold up under scrutiny — and could undermine trust in science

Published January 24, 2021 10:00AM (EST)

Microbe structures on ALH84001 meteorite (Nasa)
Microbe structures on ALH84001 meteorite (Nasa)

This article originally appeared on Undark.

Britons who switched on their TVs to "Good Morning Britain" on the morning of Sept. 15, 2020, were greeted by news not from our own troubled world, but from neighboring Venus. Piers Morgan, one of the hosts, was talking about a major science story that had surfaced the previous day, informing his viewers that "there may be some form of life on Venus."

Astronomers, he reported, were considering that "living organisms may be floating around in the clouds of planet Venus." He was then joined, via live TV link-up, by Sheila Kanani, a planetary scientist and outreach officer with the Royal Astronomical Society (RAS). Morgan put it to her point-blank: "Is there life on Venus?" Kanani replied diplomatically but enthusiastically: "We can't definitively say that there's life on Venus at the moment. But whatever is going on at Venus is very exciting indeed."

The research, which had been published the previous day in the journal Nature Astronomy by an international team of scientists, claimed that observations made with the James Clerk Maxwell Telescope in Hawaii and the Atacama Large Millimeter Array (ALMA) in Chile had detected the chemical phosphine, identified by its spectral signature, in the atmosphere of Venus, and that this could be read as a possible sign of life on the cloud-covered planet. Media outlets around the world carried the story — it made the front page of The New York Times — and tens of thousands tuned in to a press conference RAS co-organized to hear the scientists themselves discuss the finding. (Video of that event has by now racked up more than a quarter of a million views on YouTube.)

It was, in brief, the big astronomy story of 2020 — or at least it was poised to be, if the results held up. Within weeks of the initial publication, however, doubts surfaced. Some astronomers questioned the methodology behind the data analysis; it's possible, they argued, that the purported signal wasn't due to phosphine at all, but rather due to sources in the Earth's atmosphere or possibly in the telescope itself. Another team of astronomers re-analyzed some of the data and concluded there was "no statistically significant detection of phosphine."

By Nov. 20, the journal's editors had appended a cautionary tag to the article: "The authors have informed the editors of Nature Astronomy about an error in the original processing of the ALMA Observatory data underlying the work in this Article, and that recalibration of the data has had an impact on the conclusions that can be drawn."

Meanwhile, even if the team really had detected phosphine, there was no way to be certain of its biological origin; the paper's authors acknowledged this, merely noting that on Earth, phosphine is typically associated with micro-organisms, but allowing that it could be due to some unknown chemical process. For many who heard the news, though, it was all too easy to leap from somewhat ambiguous spectral lines to little floating creatures in the Venusian atmosphere.

The kinds of astronomy and physics "breakthroughs" that generate breathless media coverage on par with the Venus-phosphine story seem to come along at regular intervals. Readers may recall the purported detection of signs of primordial gravitational waves from the early universe in 2014; claims of neutrinos moving faster than light in 2011; the supposed discovery of bacteria that can use arsenic in place of an element considered vital to life in a California lake in 2010 — and the grandest such claim of the last 25 years, the alleged discovery in 1996 of fossilized micro-organisms on a Martian meteorite that had been recovered in Antarctica. (That claim was so astounding that it prompted a speech by then-President Bill Clinton.) In the end, none of these claims have held up.

On the other hand, numerous other stories, equally big, have held up: In 2012, physicists used the Large Hadron Collider at CERN to confirm the existence of the Higgs boson; and, less than two years after the claimed detection of primordial gravitational waves, physicists used the Laser Interferometer Gravitational-Wave Observatory (LIGO) detectors to record gravitational waves emitted by merging black holes.

No one begrudges the attention bestowed on either of those discoveries, both of which have been recognized with Nobel Prizes. And hype can certainly be found in other fields; the human genome project understandably generated an enormous amount of media interest, as have various controversies over cloning. But astronomy and physics, which offer glimpses of the farthest reaches of the universe and perhaps shed light on ancient questions about our place in the cosmos, seem to trigger a never-ending stream of provocative pronouncements. And all too often those claims seem to fall flat.

In science, new findings face intense scrutiny. That is, after all, how science is supposed to work, and it's hardly surprising that some claims turn out to be wrong. But if claim after claim fails to live up to the hype that surrounds it, scientists worry that the public will feel let down, and may even question whether scientists can be trusted — and whether they deserve to be funded. In other words, hype has consequences, and public trust in the scientific enterprise is at stake.

And yet the scientists and journalists that I spoke with for this piece are hesitant to place the blame on any one part of the process. Rather, it seems the machinery of hype depends equally on those who are engaged in science, those who employ them, those who fund them, and those who report on their findings.

"There's something I call the press-academic complex," says Brian Keating, a physicist at the University of California, San Diego. "You have a mostly virtuous cycle, where academics, scientists are doing research that's fundamentally important, and then at some point, someone decides to go to their local press office." Pretty soon, local media get wind of the discovery, then national media. "At a certain point, the scientist is guaranteed to lose control of the narrative," he says.

Charles Seife, a veteran science journalist who teaches science writing at New York University, has seen the hype machinery gradually ramp up over the course of his career. "In the past 20 to 30 years, scientists have gotten a little bit more comfortable — either through social media more recently, but even previous to that, pushed by publicity-hungry administrators — to hype their own results beyond what would ordinarily be seemly or accepted by peers," he says. The pressure is not just on the scientists, but on the journalists and the various intermediaries as well; just as scientists compete for funding and prestige, journalists compete for clicks and page-views.

"When you're trying to get a story published, there is a huge pressure to make it sound like a big deal," says Natalie Wolchover, a science journalist and senior writer and editor at Quanta Magazine.

Funding agencies, meanwhile, earn bragging rights when a project they enabled makes a major breakthrough; the same goes for the institutions that employ the scientists, whether it's a university or a government agency such as NASA.

"Everyone has skin in the game," says Seife. "Everyone benefits from having something get a lot of publicity and a lot of attention — presuming it holds up."

* * *

Keating has had something of an insider's view of the hype machine. He co-developed the telescope known as BICEP (Background Imaging of Cosmic Extragalactic Polarization) — predecessor to BICEP2, which made news in 2014 by revealing what was said to be evidence of ripples in space-time known as gravitational waves, or rather, the imprint that those waves left on the cosmic microwave background radiation, an all-sky glow left over from the early universe. If those gravitational waves from the early universe had truly been found, it would lend support to a theory known as cosmic inflation, an element of the Big Bang model of the early universe.

It also would have been a Nobel-worthy discovery. Indeed, Keating's book about his experiences as a cosmologist, including the BICEP2 project, is titled "Losing the Nobel Prize." As it turned out, the signal that BICEP2 measured was largely the result of dust in our own Milky Way galaxy, and not a signature of early-universe physics. (The waves successfully detected two years later by the LIGO facility were registered directly, rather than via any effect on the cosmic microwave background.)

In the six years since BICEP2's purported discovery, Keating has come to realize that publicity is as much a part of his field as telescopes and grant applications. Major findings in astronomy and physics now routinely include press conferences. On the surface, a press conference makes perfect sense: It brings scientists and journalists together in one room (or, in Covid-times, a single webinar or Zoom screen). If the journalists have questions, the scientists can answer them in real time. But some scientists feel the press conference is a bad idea — especially if the findings have not yet been published in a peer-reviewed journal (as was the case with BICEP2; the research wasn't published until a few months later).

Scientists who present their findings to the press before sharing their work with their peers are jumping the gun, says Marcelo Gleiser, a physicist at Dartmouth College. "And that, to me, is a capital sin." This, he says, was BICEP2's big mistake. "They did a good experiment — but they did not wait," he says. "They wanted to make a big splash."

But, Keating notes, the BICEP2 results weren't kept secret, either, having been posted to arXiv.org — a sort of digital clearinghouse for research in physics — on the same day as the press conference. In his book, he explains the team's decision to crowdsource the vetting of their work: "Instead of restricting our findings to a single referee's eyes, which is typically what happens when scientists submit their findings to an academic journal — one who might well be a competitor and leak our results — we opened it to the whole world." He notes that other research teams had adopted the same strategy, so they believed there was "strong precedent" for their course of action.

Today Keating feels differently. Having a press conference "obviously, in retrospect, was a big mistake," he says. In fact, he now sees press conferences as "a spectacle that science doesn't need," noting that they were rare until the 1990s. A scientific breakthrough would have the same impact with or without a press conference, he says. Plus, if you're shown to be wrong, "you have to walk back the result and somehow put the toothpaste back in the tube."

As much attention as the BICEP2 press conference got, the highly polished YouTube video released by Stanford University — one of several institutions that supported the research — drew far more eyeballs. In the video, a researcher named Chao-Lin Kuo, who had designed the detectors at the heart of the BICEP2 experiment, walks up to the house of theoretical physicist Andrei Linde, one of the founders of inflation theory. Kuo, champagne in hand, tells Linde the telescope has found a clear signal of those primordial gravitational waves. Linde is ecstatic; the cork on the champagne is popped; tears well up. The video has been viewed more than 3 million times. The video was memorable, says Gleiser, but given how the story eventually played out, he now sees it as misguided. "It is embarrassing," he says. "It is bad for everyone's reputation, in the end."

For Wolchover, the BICEP2 case and the discovery of gravitational waves announced by the LIGO team just two years later make an interesting contrast. In both cases, there was a much-watched press conference — but in the case of LIGO, the published, peer-reviewed article was made available at the same time as the news briefing. With BICEP2, there was voluminous media coverage but little scientific scrutiny, since the research had yet to be published. This ultimately "led to this very public downfall for that experiment, and egg on the face of some of the people covering it," she says.

And yet, peer review is no panacea; the Venus-phosphine paper had in fact been peer-reviewed at the time the results were presented to the press. The key, Wolchover says, is skepticism — something she believes was lacking in media coverage of the Venus story. She fears that people will be left with "some vague idea that we discovered life," she says. "And then they won't see next week's story that's buried at the bottom of the newspaper, if it even makes it in somewhere like [The New York Times] saying that that result has been questioned." A few weeks after the story broke, she tweeted: "The claim should have been approached with massive skepticism, given minor billing, or been skipped altogether for now."

* * *

Marcia Bartusiak, a science journalist with decades of experience and an emeritus professor in the graduate science writing program at MIT, has seen it all before. For the scientists, there is "that desire to perhaps stick your neck out a little farther than you should have," she says. "They're on a tightrope of: They want the public's interest, they want the continued funding — but they have to be careful to not disillusion people."

Journalists, meanwhile, face similar pressures. Early in her career, Bartusiak was reporting for Discover magazine on the purported discovery of Martian meteorites. "But when I wrote the story, and I contained both sides, the editors wanted to pump up the exciting part — you know, 'Meteorites from Mars?' And they wanted to take out all the stuff about the evidence against it because they said 'Oh, that just dilutes the story. It dilutes the punch.'"

About a decade later, Martian meteorites were in the news once again, this time with the startling claim that fossilized micro-organisms had been detected on a particular 2-kilogram chunk of rock known as Allan Hills 84001, named for the region of Antarctica where it had been recovered. Before the NASA press conference, held in Washington, D.C. on Aug. 7, 1996, the scientists were likely urged to "be a little bit more firm, be more emphatic," Seife found in his reporting after the event. The push to be confident rather than cautious and reserved was clear, he says. Soon afterward, President Clinton spoke from the south lawn of the White House, pledging to fully support "the search for further evidence of life on Mars."

Eventually, the claims were scaled back; the scientific consensus, when it was eventually reached, was that the rock most likely contained no micro-fossils after all. When I asked Seife how the "no fossils" coverage compared to the initial reporting, he laughed. The story "quietly faded away," he said.

In the case of the Venus story, however, not everyone views what happened as problematic. "I don't see it as an example of something that was horribly overhyped and then went south," says David Grinspoon, an astrobiologist at the Planetary Science Institute in Tucson, Arizona. For starters, he says the team was reasonably cautious in presenting their results. If "other people show that they made a mistake, maybe that'll end up being the story. That's not a horrible story for science. That just shows how it works," he says. And even if the results are mistaken, he says, it could be a "useful mistake" if the episode drives more scientists to investigate Venus's atmosphere.

Just as the Venus-phosphine story was fading from the headlines, another seemingly big space story broke: In late October, NASA announced that astronomers using an airborne infrared radio telescope known as the Stratospheric Observatory for Infrared Astronomy had detected water on the sunlit side of the moon, in a large lunar crater known as Clavius. Previous observations had been ambiguous, but now the scientists said they were sure. As NASA press releases go, this one was cautiously worded, noting that even the Sahara Desert contains 100 times more water than SOFIA had detected. Even so, it became a huge story. NASA's administrator, Jim Bridenstine, tweeted that, while it wasn't clear if could serve as a practical resource, "learning about water on the Moon is key for our #Artemis exploration plans," referring to NASA's plan to land humans on the moon by 2024.

But, as Seife notes, we've known there's water on the moon ever since the Clementine mission in the mid-1990s. NASA, he says, took a moderately noteworthy discovery "and all of a sudden it turned to, 'We're going to land astronauts there, and they'll harvest the water, and launch rockets up from the water' — it just makes no sense." In a similar vein, Phil Plait, an astronomer and prolific science blogger, tweeted that the published paper is "very interesting and cool scientifically but tying it to Artemis is a MAJOR reach. Like, no. Stop."

* * *

Several of the people I spoke with described a kind of feedback loop in which scientists are tempted to over-inflate their claims, with journalists playing along for the sake of a compelling story — with no obvious way of breaking the cycle. "I don't know if we can totally abolish the hype," says Bartusiak. "I think it's always going to be with us." An obvious danger, notes Gleiser, is that the public could become jaded, especially if science journalism begins to parallel the seesaw-like stories sometimes seen in health and lifestyle reporting, in which coffee, chocolate, and wine are either good for you or bad for you, their efficacy seeming to depend on the day of the week. The risk, Gleiser says, is that "we lose this very precious thing that our ancestors have worked very hard to develop, which is trust."

A second, related, danger is that with everyone shouting their findings at the greatest possible volume, nothing coherent can be heard above the din. "It's like how in a restaurant, when people start talking loudly, then other people start talking louder, and eventually everyone's screaming," says Wolchover.

A good first step, she and others suggest, would be to encourage coverage that more closely reflects the significance of the research being put forward. When that research is inconclusive, the audience needs be told so.

"If the public's trust in science is undermined, that has a devastating impact, not only on scientists," says Keating. "First the scientists will suffer, but then society will suffer." This is especially serious, he suggests, in an age when trust in science and scientists is already on shaky footing. People will think, "We can't trust science, which means knowledge, then who can we trust?"

This article was originally published on Undark. Read the original article.


By Dan Falk

MORE FROM Dan Falk


Related Topics ------------------------------------------

Astronomy Media Industry Science Undark