Questionable Numbers for a Questionable Remedy

Questionable Numbers for a Questionable Remedy

Echinacea might be useful as a cold remedy or preventative, but science hasn't shown it yet

When you first feel the sniffles and wonder what to grab from your medicine cabinet, perhaps you should first check some numbers. Especially if one of your choices is echinacea.

The evidence on whether echinacea helps fight colds has been confusing. A 2005 study concluded that the herb doesn't help, but a few months ago, a team of pharmacists claimed that it decreases the odds of getting a cold by 58 percent and reduces the length of colds by one to four days.

Scientists continue to debate whether echinacea helps fight colds effectively. iStockphoto

The evidence favoring echinacea is not as rosy as the second study might suggest, however. For one thing, the statement that echinacea decreases the odds of getting a cold by 58 percent is based on terminology that can be misleading. Second, that study was a meta-analysis, a compilation of previous studies. It was a less rigorous and less tightly controlled analysis than the first. Although that meta-analysis may suggest that echinacea shouldn't be dismissed entirely as a cold remedy, it certainly doesn't prove that the herbal remedy works.

For practical purposes, people want to know whether they're less likely to suffer from a cold if they take echinacea, and if so, by how much. It's not hard to compute that from data in the study. Patients who didn't take echinacea got a cold about 65 percent of the time, whereas those who did got sick about 45 percent of the time. So people who took echinacea had about a 30 percent smaller chance of getting a cold (^{65 – 45}/₆₅).

So why does the study conclude that echinacea reduced the odds by 58 percent rather than 30 percent? It turns out that in ordinary speech, "reducing your odds" means the same thing as "reducing your chances," but in statistics, their meanings are quite different.

Statisticians define the "odds" as the ratio between the chances that something will happen to the chances that it won't. Although the term has infiltrated our vocabulary, we don't tend to think about odds much in regular life, outside of a gambling context. We keep it simpler and just talk about the chances that something will happen, rather than the ratio between the chances that it will happen and the chances that it won't.

According to the data from the study, the reduction in the odds of getting a cold if you take echinacea turns out to be almost twice as high as the reduction in the chance of getting a cold if you take echinacea. Seeing why requires a bit of arithmetic. If you hadn't been taking echinacea, you would have had about a 65 percent chance of getting a cold and about a 35 percent chance of not getting one. That means that the odds of getting a cold without echinacea would be just under 2:1 (to be precise, the chances would be 65.23 percent, so the odds would be 0.6523/0.3477, or 1.88).

If you were taking echinacea, you would have had about a 45 percent chance of getting the cold and about a 55 percent chance of beating it. So the odds of getting a cold would have been just over 4:5 (to be precise, the chances would have been 44.87 percent, so the odds would have been 0.4487/0.5513, or 0.81). That means taking echinacea would have decreased the odds from 1.88 to 0.81, or by 58 percent (^{1.88 – 0.81}/_1.88).

 

Colds cost the U.S. economy 40 billion dollars a year. iStockphoto

 

Of course, most people would be delighted to have 30 percent fewer colds. But there are reasons to suspect that echinacea may not really be able to accomplish that. Understanding those reasons requires unraveling some details of how the two studies were done.

In the 2005 study, Ronald B. Turner of the University of Virginia School of Medicine and his colleagues did the kind of work one typically imagines in a medical study: They randomly divided the patients into two groups, gave one group echinacea and the other a placebo, exposed all the patients to the cold virus, and monitored them to see how many in each group caught a cold. Neither the doctors nor the patients knew who was receiving echinacea.

The researchers found that the patients who had taken echinacea were a tad better off, but the effect was so small that it might well have been caused by random variations. In technical terms, the effect was not "statistically significant."

The later meta-analysis was an entirely different sort of project. Instead of putting virus-infected cotton swabs in patients' noses, Dr. Craig I. Coleman of the University of Connecticut School of Pharmacy and his colleagues compiled data from previous studies and did some math.

The idea of a meta-analysis is to combine the data from a bunch of small studies into a single large study. The advantage of a big study is that it isn't impacted by random effects as much as a small one, just as a big ship doesn't rock in the waves as much as a dinghy. Randomness affects the results of studies because individual responses to drugs vary: for example, a certain antihistamine might work well for your allergies but do nothing for your friend's. The smaller the number of patients enrolled in a study, the higher the effect of individual variations on the aggregate data. As a result, in a small study the researchers can only be sure that the treatment actually did any good if two or more groups show a large difference in response.

With only 250 participants in his study, the small differences Turner saw were statistically insignificant. If he had seen the same effects with 10,000 patients, it would have been clear that echinacea, not chance, was responsible.

But Turner is not the only researcher who has studied echinacea. Coleman and his colleagues found 13 other, similar studies. By pooling the data together in a meta-analysis, they tried to see if echinacea had a statistically significant impact when studied with a larger population. They concluded that it did, both for prevention and for treatment.

Although meta-analyses can be revealing, many researchers and physicians view them with suspicion because they can go wrong in many ways. The echinacea meta-analysis has been criticized for having many such difficulties.

For one thing, in real life, no two studies are ever done in exactly the same way. They use different dosages, different types of patients, and different regimens. The variation is even greater with herbal remedies, where the variety of plant and the way it is prepared can vary dramatically.

"You only can use more-or-less, reasonably comparable studies" in a meta-analysis, says Klaus Linde of the Technical University in Munich, who published a systematic review of echinacea studies. "In our opinion, this was not the case in this study."

Furthermore, some studies are simply better done than others, and including poorly designed studies can skew the results of a meta-analysis. "I think there's a general consensus that many of the early studies of natural remedies were done using study designs and methods that probably are not of the same quality that we would like to see for standard pharmaceutical-type drugs," Turner says. "In my opinion, some of the studies that they chose to include that were large and had a lot of weight in the analysis weren't designed very well."

Coleman says that his group included as many relevant studies as they could find in an attempt to include as much information as possible. He acknowledges that studies with quite different designs were included, but he points out that they conducted sub-analyses that examined only particular types of studies. "Other analyses are perhaps less prone to potential error than ours is," Coleman says, "but ours is less prone to miss something."

Coleman argues that dismissing echinacea on the basis of Turner's study is premature, particularly given that "our meta-analysis shows that the preponderance of the evidence suggests that it may work." He points out that Turner used one species of echinacea, Echinacea angustifolia, when the more commonly used species is Echinacea purpurea, which may be more effective. He also argues that the dosages Turner used were too low to show an effect. "Ultimately, what we're trying to suggest is that more studies should be done."

Turner says he chose Echinacea angustifolia because it was the species originally used as a medicinal in the late 1800s, and he says that there is little reason to think that the two species would act differently. He also says that he used the highest dosage considered safe at the time.

Although one can always imagine other variations that might be tested, Turner says that enough is enough. "Given the available data, we believe it is most reasonable to conclude that echinacea is not useful as a treatment for the common cold," he wrote in a letter to the New England Journal of Medicine. "This conclusion should stand until those who promote, manufacture, and sell these products produce convincing evidence of a clinically meaningful benefit." He is optimistic, however, that as herbal medicines continue to be studied carefully, some of them will prove to be effective.

Linde, despite his skepticism of Coleman's meta-analysis, argues that more research is needed. He says that some studies tentatively suggest that the aerial parts of Echinacea purpurea may shorten colds if taken early in the course of the illness. Furthermore, he points out, people continue to use echinacea frequently, spending $126 million on the herb in the U.S. alone in 2006. It would be nice to know for sure whether they're getting any benefit for their money.