Adventures in bad science reporting -- Cancer meat edition

Recently the International Agency for Research on Cancer (IARC), a research arm of the World Health Organization, classified processed meat as a known carcinogen and red meat as a probable one. Some people took it as a strong argument against meat. Industry PR called it "dramatic and alarmist overreach." I think this is an example of how poor science reporting, from the original press release to mainstream and secondary follow-ups, plus public innumeracy, can lead to over-political science news.

I.


Let's start at the source: What did the IARC's press release [pdf] actually say?
After thoroughly reviewing the accumulated scientific literature, a Working Group of 22 experts from 10 countries convened by the IARC Monographs Programme classified the consumption of red meat as probably carcinogenic to humans (Group 2A), based on limited evidence that the consumption of red meat causes cancer in humans and strong mechanistic evidence supporting a carcinogenic effect.

This association was observed mainly for colorectal cancer, but associations were also seen for pancreatic cancer and prostate cancer.

Processed meat was classified as carcinogenic to humans (Group 1), based on sufficient evidence in humans that the consumption of processed meat causes colorectal cancer.

[...]

The experts concluded that each 50 gram portion of processed meat eaten daily increases the risk of colorectal cancer by 18%.
Emphasis theirs. Also, for American reference, a quarter-pound (4oz.) is about 113 grams. And wow! 18% increase in risk of colorectal cancer! Classification into Groups! That sounds scary! There's enough evidence that we might as well call "red or processed meat" cancermeat for short.

And now, let the skeptical caveats begin:

First, the full IARC study is to be published in Lancet Oncology, which is about cancer research and not about health policy.

Second, just from the announcement we don't know exactly what these Groups mean. I know my very first connection was with the United States' scheduling policy for drugs, where Schedule I means "way too dangerous and addictive for any use" and Schedule II means "maybe sometimes medical use," or something like that.

Third, we don't know the base rate of colorectal cancer! Or any rates, really. Just this "eating cancermeat will increase your risk of cancer by 18%" figure. Needless to say, base values matter when you're talking about proportions: (1.18) * (0.01) is a lot different than (1.18) * (0.20), for example.

II.

So how did Facebook see this? That is, how did the mainstream media cover this? Let's start at the Washington Post, a widely-read mainstream newspaper. Hot dogs, bacon and other processed meats cause cancer, World Health Organization declares, or so the title goes. This was posted to Wonkblog, so it's I guess not full journalism? But "wonk" is supposed to mean "getting nerdy about details" so I think we can still hold them to a high standard.
A research division of the World Health Organization announced Monday that bacon, sausage and other processed meats cause cancer and that red meat probably does, too.
As a first paragraph goes, that's pretty fair.
The report by the influential group stakes out one of the most aggressive stances against meat taken by a major health organization, and it is expected to face stiff criticism in the United States.
Ding ding ding! Politicization right off the bat. "One of the most aggressive stances against meat"? This is cancer research, not policy recommendation!
But the panel’s decision was not unanimous, and by raising lethal concerns about a food that anchors countless American meals, it will be controversial.

The $95 billion U.S. beef industry has been preparing for months to mount a response, and some scientists, including some unaffiliated with the meat industry, have questioned whether the evidence is substantial enough to draw the strong conclusions that the WHO panel did.
The Magical Balance Fairy is strong with this one. There are a couple times in this piece where the meat industry either flatly denies the conclusion, or plays a "well life causes cancer, so nyah!" kind of game. (Still, I don't think people who oppose the meat industry have such a powerful argument in this IARC announcement, as I'll explain later.)
In reaching its conclusion, the panel sought to quantify the risks, and compared to carcinogens such as cigarettes, the magnitude of the danger appears small, experts said. The WHO panel cited studies suggesting that an additional 3.5 ounces of red meat everyday raises the risk of colorectal cancer by 17 percent; eating an additional 1.8 ounces of processed meat daily raises the risk by 18 percent, according to the research cited.
This is sort of the Big Important Thing To Keep In Mind throughout all of this, and yet it's not even quantified—ironic since the beginning clause of this sentence is about seeking to quantify risks. But in terms of bad reporting it doesn't hold a candle to what comes after:
For consumers, the WHO announcement offers scant practical advice even while casting aspersions over a wide array of foods. [...] How much of those is it safe to eat? The group doesn’t offer much guidance[...] Should we be vegetarians? Again, the group does not hazard an answer.

And how exactly does red meat and processed meat cause cancer? The group names a handful of chemicals involved in cooking and processing meat, most of them nearly unpronounceable, and some believed to be carcinogenic. [...] Despite the voids in the science, the WHO findings might cast a pall over diners and those who serve them.
This is hack science journalism. That's to say nothing of the ridiculous related headlines inserted in the body of the article, like "WHO says hot dogs, bacon cause cancer. Does this mean we should all become vegetarians?" and "95 percent of the world's people may be wrong about salt," in a bid to out-clickbait Clickhole.

It's worth noting at this point that the article's author, Peter Whoriskey, is not a science writer for WaPo. His bio says he's mostly does finance and economics writing, with a bit about pharmaceuticals. I guess he was picked for this because the WHO? Anyway, the article devolves into some consumer-at-the-Whole-Foods quotes and then two nice bits from industry spokespeople.
“We simply don’t think the evidence supports any causal link between any red meat and any type of cancer,” said Shalene McNeill, executive director of human nutrition at the National Cattlemen’s Beef Association.
I don't really fault the meat industry people for saying this sort of thing. Human psychology being what it is, and innumeracy besides, people might just avoid something altogether if a scientist-type person (or just someone in a white labcoat) tells them that the thing "has a risk" or "puts them at risk." As Han Solo says, "never tell me the odds!"

So it's a little game. Scientists say "such and such thing increases risk by X percent, error bars are thus and so" (oh wait this is science reporting there are never error bars /s) and PR reps say "that's preposterous, there's no causal link here!" and consumers are relieved that doing such and such won't literally turn them into a giant tumor but maybe end up moderating their behavior over time.
The WHO panel “says you can enjoy your yoga class, but don’t breathe air (Class I carcinogen), sit near a sun-filled window (Class I), apply aloe vera (Class 2B) if you get a sunburn, drink wine or coffee (Class I and Class 2B), or eat grilled food (Class 2A),” said Betsy Booren, vice president of scientific affairs for the [North American Meat Institute].
Okay, that goes a bit too far. This sort of ¯\_(ツ)_/¯ attitude towards scientific research—"well yeah, but scientist say things all the time you guys! Am I right? Scientists...!"—really grates on me. (It's the same mode of quasi-ludditic argumentation that some people levy against Common Core Math, I think. The "I didn't bother to try to understand, but fuck it" attitude.)

The overall tone of the WaPo piece is fascinating and frustrating to me. By taking the very elements of proper scientific research—reviews of meta-studies, careful statistical analysis, risk and uncertainty quantification—WaPo turns around and fits it to a narrative about scientists never being able to make up their minds about what we should eat. ("Scientists...!")

III.

Okay, but enough slagging on WaPo/Whoriskey (at least for now). What about other mainstream outlets? PBS NewsHour has two pieces about this, one the straight news report and one a FAQ about the details. The author of both, Nsikan Apkan, is a science reporter for NewsHour, for what that's worth.


The NewsHour report was the first piece I saw on this, too. Let's see how it does:
Bacon, sausage and other processed meats are now ranked alongside cigarettes and asbestos as known carcinogens, the World Health Organization announced today. Processed meats cause cancer, and red meat likely causes cancer, the health agency says in a new report.
I actually think that the opening paragraph is more sensationalized than Whoriskey's, although overall (as I'll explain) I think Apkan does a better job overall. Ranked alongside cigarettes and asbestos is a super strong phrase, without the proper stipulations about what that ranking means.

The PBS article helpfully includes links to the full Lancet Oncology article in the third paragraph, as well as a meta-study cited as a "main line of evidence," and other helpful outside websites. In sharp contrast, the WaPo article doesn't out-link at all. To get to the official press release or the Lancet Oncology article (just the abstract!), you'd have to have clicked through to a different WaPo blog post ("WHO says hot dogs, bacon cause cancer. Does this mean we should all become vegetarians?" because of course that's the title) by yet another non-science staff writer. It's more of an FAQ, and better than the original WaPo article, but still inferior to even the regular PBS article, to say nothing of PBS' own FAQ.

Moving on, we get to the first substantively problematic paragraph:
Processed meat now falls into “group 1,” meaning it ranks as high as tobacco smoking, the most dangerous variants of human papillomavirus (HPV) and asbestos exposure in terms of causing cancer. Red meat lands in “group 2A” with inorganic lead.
Combined with the provocative opening paragraph, this makes for an overall alarming tone. This piece doesn't fare much better than WaPo, but at least it doesn't go fawning over meat industry spokespeople and actually reports on how the science was done, rather than Reporting The Controversy. (Seriously, the subtitle to the WaPo piece was "U.S. meat industry objects controversial finding and prepares for battle.") The PBS piece just links to the WaPo piece in the final two paragraphs. So there's still a bit of Reporting The Controversy, but minimal. And lots of actual science reporting.

The FAQ ("Exactly what processed meat should I avoid, and other questions") is much much better, but contains what I think is a legitimate, and serious, factual mistake. It carefully goes over what the IARC means by "red" and "processed" meats, what "causes" means versus "is linked to," and provides some helpful infographics from Cancer Research UK's science blog (which I found before looking at the FAQ, and which I'll get to in a bit).

The big mistake is how the FAQ talks about increased risk.
On an individual level, it’s hard to say. On a population level, the WHO report cites this epidemiology meta analysis, which examined colorectal cancer studies going back to 1966.

Based on that study, a person who eats 50 grams per day of processed meat has an 18 percent chance of developing colorectal cancer. A person who eats 100 grams has a 36 percent chance and so on. According to Cancer Research UK, 50 grams per day would be on par with two slices of ham.
Yikes! That's not what the study says! Even the highest rate of colorectal cancer is nowhere near as high as 18 percent. That's a bad mistake.

IV.

Let's take a look at the 2011 meta-study cited by the IARC and linked to (twice!) by PBS. It's called "Red and Processed Meat and Colorectal Cancer Incidence: Meta-Analysis of Prospective Studies," by Chan et al. Here's the abstract (full text just continues below, yay for PLoS One!):
Red and processed meats intake was associated with increased colorectal cancer risk. The summary relative risk (RR) of colorectal cancer for the highest versus the lowest intake was 1.22 (95% CI  = 1.11−1.34) and the RR for every 100 g/day increase was 1.14 (95% CI  = 1.04−1.24). Non-linear dose-response meta-analyses revealed that colorectal cancer risk increases approximately linearly with increasing intake of red and processed meats up to approximately 140 g/day, where the curve approaches its plateau.
Finally, some more numbers to work with! What did I notice?
  • First, the numbers given are relative risk. I'm not sure what that means vis-a-vis population statistics and actually getting cancer, so I'll have to look it up.
  • Next, the relative risks given seem small: 1.22 for summary RR and 1.14 for each 100g/day increase. But, given that I'm ignorant of relative risk, I can't say whether that these numbers are actually small relative to other known carcinogens or what. Maybe tobacco smoke has a RR of 1.01 for lung cancer! (Spoiler alert: it doesn't, that number is much higher.)
  • The 95% confidence intervals (CI) for the RRs are pretty tight, just plus or minus 0.1. That seems pretty definitive. Given that it's "relative" risk I would expect the null hypothesis to be that the RR is 1.0, so for summary RR that would be like 3 standard deviations away. A strong result indeed!
  • The relative-risk curve approaches a "plateau" at 140 g/day, which makes the "and so on" part of the PBS FAQ seem rather absurd.
Relative risk, according to Wikipedia and the Mayo Clinic, is just the ratio P(has colorectal cancer, given that) / P(has colorectal cancer, given that doesn't eat red meat). Or, in other words, the proportion of cancermeat-eaters who have colorectal cancer, divided by the proportion of non-cancermeat-eaters who have colorectal cancer. The Mayo Clinic article has a couple important paragraphs. First:
For instance, compare the relative lung cancer risk for people who smoke with the relative lung cancer risk in a similar group of people who don't smoke. You might hear relative risk being expressed like this: The risk of lung cancer for men who smoke is 23 times higher than the risk for men who don't smoke. So the relative risk of lung cancer for men who smoke is 23.
Twenty-three, not one-point-something. Seems important. (In percentage-higher terms, it's 2300% for smoking versus 18% for worst-case cancermeat.) Moreover:
Risk seems greater when put in these terms. A 100 percent increase in risk may seem enormous, but if the risk began as 1 in 100 people, a 100 percent increase in risk means that 2 out of 100 will be affected.
Right! So what are the base rates? The meta-analysis doesn't say, but that's not really an issue. The aim of that analysis was to show a clear statistical link, not tell us anything about base rates. As far as classification goes, we only care about solid statistical evidence, not the magnitude of the effect.

For example, if you roll a twenty-sided die, you know that there's a 5% chance the face will come up, say, on an 8. You know this with a very strong certainty. If we wanted to classify polygonal dice, say Category 1 dice are those which have a definite chance of showing an 8, and Category 2 dice are those dice which don't have a definite chance of showing an 8, then we'd have the following partition:
Category 1: d20, d12, d10, d8
Category 2: d6, d4
Is it good or bad that a d20 is Category 1 in this case? I dunno, maybe we want to roll high, in which case it's good; but maybe this is 1st edition AD&D where you try to roll low, and your character has a Wisdom score of 8 and is trying to write an article about the Greyhawk Health Organization's latest announcement about bat guano...

V.

Before I get to the Cancer Research UK blog, let's take a quick moment to appreciate that a Gawker blog did a better job at science reporting than other mainstream outlets. Yep, George Dvorsky over at Gizmodo ("Here's Why You Shouldn't Panic Over Processed Meats Causing Cancer") does it right.
It’s well known that certain meats have an association with cancer; in this respect, the latest report, which now appears at The Lancet, offers very little that is new. It merely brought the existing literature together in a way that finally allowed scientists to make some definite proclamations about the cancer risks of eating processed and red meats.
Ding ding ding ding! This is what the Washington Post article completely fails to appreciate, much less convey. "A bunch of experiments and research has been done over the years, now we've analyzed it and yup, there's something significant to the reported effects" is exactly how science moves along. It's not "woah, what's up with these scientists making proclamations and stuff?"
Specifically, the researchers say that risk of colorectal cancer increases by as much as 18% with each 50 gram (1.8 ounce) portion of processed meat eaten daily, and increases by some 17% with each 100 grams (3.5 ounces) of red meat consumption. But it’s important to keep these figures in perspective.
Indeed! Now we'll jump over to Cancer Research UK, which Dvorsky links to and quotes from (and pulls infographics from) at this point.

I found the Cancer Research UK post ("Processed meat and cancer – what you need to know") on the second page of my Google search results (search terms "red meat cancer" or something like that), after the Google Scholar citations, the Google News and main search results with clickbait-y headlines from mainstream outlets. Google should really do a better job elevating dedicated science sites when people search for science headlines.

Casey Dunlop (role at the blog unknown; but their previous post was also about bowel cancer) explains the relative risk quite nicely:
But before we move on, let’s be clear: yes, a prolonged high-meat diet isn’t terribly good for you. But a steak, bacon sandwich or sausage bap a few times a week probably isn’t much to worry about. And overall the risks are much lower than for other things linked to cancer – such as smoking.

[...]The results showed that those who ate the most processed meat had around a 17 per cent higher risk of developing bowel cancer, compared to those who ate the least.

‘17 per cent’ sounds like a fairly big number – but this is a ‘relative’ risk, so let’s put it into perspective, and convert it to absolute numbers. Remember these are all ball-park figures – everyone’s risk will be different as there are many different factors at play.

We know that, out of every 1000 people in the UK, about 61 will develop bowel cancer at some point in their lives. Those who eat the lowest amount of processed meat are likely to have a lower lifetime risk than the rest of the population (about 56 cases per 1000 low meat-eaters).

If this is correct, the WCRF’s analysis suggests that, among 1000 people who eat the most processed meat, you’d expect 66 to develop bowel cancer at some point in their lives – 10 more than the group who eat the least processed meat.
Yeah, not much to freak out about, but a fact to keep in mind when doing risk assessment. On that score, Dunlop does what nobody else (except Dvorsky, who also calls out the Guardian for a false assertion) and clarifies the point of the IARC's classification!
Processed meat has been classified as a ‘definite’ cause of cancer (or Group 1 carcinogen) – the same group that includes smoking and alcohol. And red meat is a ‘probable’ cause of cancer (or a Group 2a carcinogen) – the same group as shift work. While this may sound alarming, it’s important to remember that these groups show how confident IARC is that red and processed meat cause cancer, not how much cancer they cause.
It's worth paraphrasing: IARC classification is about whether a substance causes cancer, not how much it causes cancer. Given that cancer in general involves some probabilistic cell biology (i.e., how many mutations have stacked up to screw with properly-regulated cell replication) that's actually quite an important job. The "carcinogenicity" can be determined later.

Oh, and in further comparison to smoking:
In 2011, scientists estimated that around 3 in every hundred cancers in the UK were due to eating too much red and processed meat (that’s around 8,800 cases every year). This compares against 64,500 cases every year caused by smoking (or 19 per cent of all cancers).
So 3% of cancer in the UK is from cancermeat. Meanwhile 19% of UK cancer comes from, well, "cancer-sticks." Therefore cigarettes are something like six times more carcinogenic than red meat, but the IARC is equally confident that both these things cause cancer.

VI.

Finally, here's why political partisans shouldn't jump on this announcement too strongly.

For the meat industry, other than the perfunctory and performative denials, they should stay away from attacking the result too strongly, because it's not a very impactful result aside from all the (misinformed, miscommunicated) hype. The auto industry doesn't scoff at claims of people getting killed in car accidents. We know that driving increases one's risk of injury or death—human brains can't easily deal with controlling a thousand-pound metal frame at speed—but people still drive. They just exercise some (some) care while doing so.

For anti-meat activists, well, the ill-advisedness sort of depends on the strategy of the campaign. If it's more a "nobody should eat meat, full shop" kind of activism, then the same rules apply as to the meat industry. The result just isn't impactful enough: if people aren't convinced by the known yearly statistics on traffic deaths and severe injuries, they're not going to be convinced by low RR and low base rate. Even if they're statistically literate.

If you're more of a "humanely farmed meat is okay, in small amounts, but fuck the meat industry" person, okay, but there are many angles to gradualist animal activism. For example, in terms of "how many animals' lives are negatively affected by my consumption of meat," beef is way better than chicken, because a single cow produces lots of beef, whereas chickens produce less meat. Therefore one's yearly consumption of individual chickens is much higher than one's yearly consumption of cows. Hence, going off the IARC report alone doesn't actually do as much good as one might want, if the person you've convinced to change their lifestyle is just replacing beef with chicken.

So, here's a takeaway summary of the IARC announcement:
  •  Processed meats do cause cancer. Red meat probably causes cancer.
  • The relative risk is low: 17-18% higher than the least at-risk population.
  • The base rate of colorectal cancer is relatively low: 5.6% among low meat-eaters, and 6.6% among those who ate the most cancermeat (6.1% overall).
  • The rate of cancers from meat consumption is low relative to other carcinogens: 3% versus 19% from smoking, according to one UK study.
  • There is no real "safe" minimum for these meats, but given the low base rate, a moderate intake will not substantially increase one's lifetime risk of colorectal cancer.
Meanwhile here's a takeaway of the reporting on the announcement:
  • Washington Post sucked, gave inordinate time to industry naysayers and Reporting The Controversy.
  • PBS NewsHour barely passed, clearly a better effort than WaPo but with some frustrating alarmism in the main article and a serious error in the FAQ.
  • Freakin' Gizmodo did the best among popular outlets.
  • Cancer Research UK was MVP, being sourced from both the NewsHour FAQ and Gizmodo.
Overall it's a classic case of mainstream mediocrity (or outright awfulness) in science reporting. But at least it didn't give me cancer.