February 2001
Using Statistics to Make Regulatory Decisions
by Steven Milloy
Statistics are to governments what paint is to an artist. Statistics are the life support system for government policy—as well as the agenda of many advocacy groups whether on the left or the right. Few government policies or causes are compelling without numbers.
Take secondhand smoke, for example. This issue has two main proponents. There are those who find secondhand smoke offensive, and there are those who think that no one should smoke.
As to those who find smoking offensive, until recently, their options were to grin and bear it, or avoid smoking areas. That smoking offends some has never been a compelling enough reason for widespread smoking bans.
For the anti-smoking activist, the secondhand smoke issue is a strategy hatched in the early 1970s. Juries and governments could not prevent people from assuming the risk of smoking. But they could prevent the imposition of a risk on others.
During the 1980s, studies began to emerge reporting that women who lived with smokers—and presumably were exposed to secondhand smoke—had slightly higher rates of lung cancer. (Though these studies are themselves of questionable credibility, I won't discuss that issue now.) Although secondhand smoke reportedly was linked with lung cancer, this was not compelling enough to push for smoking bans. Why? It's simple.
Mere words are usually not as compelling as numbers. The early studies of secondhand smoke had no statistic that could be used in a sound-bite. And without a sound-bite, you're outside the window of the public's attention span and the media's willingness to provide air time.
But the US Environmental Protection Agency solved this problem. The EPA completed its own study of secondhand smoke in 1992. The studies the EPA relied on were of the same unremarkable quality as those that had previously failed to excite the public. The EPA had a trick, though.
When the EPA released it's secondhand smoke study, it didn't just claim that secondhand smoke causes cancer, it claimed that secondhand smoke caused 3,000 lung cancer deaths per year. That simple statistic—3,000 lung cancer deaths per year—was what made banning smoking a compelling argument for many.
Three thousand has also been a handy statistic for anti-smoking activists on the topic of teen smoking.
Teens have been smoking ever since they found out it was something their parents didn't want them to do. Until recently, teen smoking was regarded as a fad or phase of adolescence. When I was in high school, the school board even allowed kids to smoke in designated smoking areas on campus. Then the generation that made smoking marijuana almost as common as smoking cigarettes turned into prohibitionists. But they still needed a compelling statistic to make their case.
Everyone has heard the sound-bite "3,000 kids start smoking every day." A statistic followed on by another—"and one-third of these will die from smoking-related diseases." Where did these statistics come from?
In 1989, the Journal of the American Medical Association published a study about trends in smoking. The study estimated that about one million young persons become regular smokers every year. The study goes on to say this equates to about 3,000 young people per day becoming regular smokers.
Anti-smoking activists have translated "3,000 young people" to mean 3,000 teenagers. Ironically, the study in question did not survey teenagers—only those aged 20 or older. So the statistic about teenagers has nothing to do with them.
The other statistic—one-third of these 3,000 new smokers will die from a smoking related disease—has an equally dubious origin. The basic assumption at the heart of this statistic is that if you smoke and die from cancer or some form of heart disease, then smoking caused your death. Using this simple assumption, the anti-smoking activists in the US government have calculated that roughly 400,000 people die every year from smoking-related diseases. Combining the number of people that "die" every year from smoking with the number of new smokers is where the "one-third" statistic comes from.
The problem with the assumption that "if you smoke and die, then smoking killed you" is that death is a complex event. Smokers also tend to be heavier drinkers, have worse diets, and get less exercise than nonsmokers. Short of lung cancer, it's difficult to pinpoint the cause of death of a typical smoker.
Someone once figured out that the average lifetime heavy smoker can expect to live about six or seven years less than the average nonsmoker. A recent study in the American Journal of Epidemiology reported that smokers who exercise can expect to live only one or two years less than a nonsmoker who exercises.
Anti-tobacco types like to blame heart disease deaths on smoking. But the studies report the risk of heart disease to be about the same in smokers and passive smokers. If smoking was truly a cause of heart disease, you would expect the rate in smokers to be much higher than in passive smokers.
So most smoking statistics are pulled from thin air. They're used because they make compelling sound bites. Even if you remember no facts, you'll remember the statistic. Of all the things that can be said about smoking—it is a leading cause of statistics.
Though activists like to use statistics in advocating their causes, statistics is a two-edged sword. For years, most science and medical journals have required that study authors report the statistical significance of their results. Statistical significance is the traditional way to evaluate the strength of study results. But activists in scientists' clothing have grumbled about this requirement for years. This requirement, in large measure, prevents flaky results from being published.
Perhaps the definitive battle over statistical significance was fought when the US Environmental Protection Agency issued its risk assessment for secondhand smoke.
Traditionally, scientific results are statistically significant when we are 95 percent certain that chance or luck can be excluded as the cause of the result. This 95 percent level is not a law of nature or a rule of law, but it is the traditional level—the standard, if you will. Technically speaking, results can be statistically significant at a 50 percent level or a 99 percent level. Traditionally, though, statistical significance is achieving the 95 percent-level.
The EPA conclusion that secondhand smoke caused lung cancer was based on the result that women married to smokers—and presumably exposed to second- hand smoke—had a 19 percent higher rate of lung cancer than women married to nonsmokers. When the EPA published this result, it claimed the result was statistically significant. It wasn't—at least not in the traditional sense of statistical significance at the 95 percent level. The EPA result was only statistically significant at the 90 percent level.
What's the difference you ask? Isn't 90 percent pretty close to 95 percent? Actually, a result that is statistically significant at the 90 percent level is twice as likely to have occurred by chance as a result at the 95 percent level.
The EPA recognized the importance of achieving some sort of statistical significance. Since the EPA could not attain significance at the 95 percent level, it changed the rules of the game and opted for an unprecedented 90 percent level. The EPA knew that if it could not attach statistical significance to its result, it would not be taken seriously.
Many were shocked at this manipulation of statistical significance—including a federal judge who invalidated the EPA risk assessment of secondhand smoke in July 1998. But the EPA learned an important lesson: the requirement of statistical significance needs to go.
So when the EPA proposed its revised guidelines for assessing the cancer risks, guess what? The requirement of statistical significance was deleted. Not only was it deleted, the EPA denied deleting it.
Though the EPA's Science Advisory Board advised the EPA staff to re-instate the requirement of statistical significance, the future of this requirement at the EPA remains in doubt as the EPA has still not issued the guidelines in final form, though they were first proposed in May 1996.
But the EPA is not alone in its jihad against statistical significance. Over the last ten years or so, the number of activists in scientists' clothing has increased. Their funding by government agencies gives them the air of legitimacy. And now they have unprecedented say in the science establishment. They are no longer the fringe. They are the mainstream.
With their new-found power, these activists have set their eyes on ridding statistical significance from the medical and scientific literature. Toward this end, they have succeeded in convincing the American Statistical Association to take up the issue of statistical significance. The Association is considering whether to issue a statement that would essentially obliterate statistical significance as a requirement once and for all.
If statistical significance is removed as a requirement for evaluating scientific and medical studies, we will see an unprecedented flood of junk science.
Statistical significance is not science— there is no question about that. But then again neither are many studies published in science and medical journals. They are merely exercises in statistics. Statistical significance is how we prevent these non-scientific exercises from being wrongly regarded as science. Statistical significance is how we prevent these non-scientific exercises from being used as the basis for regulation.
Reference
Ferrucci L., et al., "Smoking, Physical Activity, and Active Life Expectancy," American Journal of Epidemiology 1999 (April 1) 149 (7): 645-53.
Steven J. Milloy is publisher of Junkscience.com and is an adjunct scholar at the Cato Institute. He holds a Master of Health Sciences in biostatistics from The John Hopkins University School of Public Health, a Juris Doctorate from the University of Baltimore, and a Master of Laws in securities regulation from the Georgetown University Law Center.
info@fraserinstitute.ca
4th Floor, 1770 Burrard Street, Vancouver, BC, Canada, V6J 3G7
Tel: (604) 688-0221 Fax: (604) 688-8539 Book Orders: 1-800-665-3558 ext. 580
You can contact us at the above email address for any comments or information requests. Please report any dead links or technical problems.