Analysis of Meta-analysis
You’ve probably seen quite a few posts regarding the Stanford study that concluded that there was little advantage to consuming organic foods.
Annals of Int Med http://goo.gl/fv4fG
Gaythia Weis asked me to comment on her post about ta blog post about he study. http://goo.gl/rerki
Sorry to disappoint you all, but I haven’t had enough time to dig into this framing debate. It really isn’t my area anyway. However, I use statistics all of the time in my research. So I’m going to introduce what is meta-analysis and hopefully return to the articles later.
Meta-analysis is a method of contrasting and combining results from a large group of studies. That sounds like an advantage and it is, i.e., strength in numbers. However, there are disadvantages and meta-analyses can be done wrong. The biggest disadvantage is that each study has a purpose and study design that is most likely not identical. So the key is how do you decide which studies to include and which studies to exclude. In today’s #ScienceSunday SSHOw, I hope to discuss this in a little bit more detail. The Wiki is actually quite good.
http://en.wikipedia.org/wiki/Meta-analysis
Getting back to the pesticide story here’s the abstract from the article.
Background: The health benefits of organic foods are unclear.
Purpose: To review evidence comparing the health effects of organic and conventional foods.
Data Sources: MEDLINE (January 1966 to May 2011), EMBASE, CAB Direct, Agricola, TOXNET, Cochrane Library (January 1966 to May 2009), and bibliographies of retrieved articles.
Study Selection: English-language reports of comparisons of organically and conventionally grown food or of populations consuming these foods.
Data Extraction: 2 independent investigators extracted data on methods, health outcomes, and nutrient and contaminant levels.
Data Synthesis: 17 studies in humans and 223 studies of nutrient and contaminant levels in foods met inclusion criteria. Only 3 of the human studies examined clinical outcomes, finding no significant differences between populations by food type for allergic outcomes (eczema, wheeze, atopic sensitization) or symptomatic Campylobacter infection. Two studies reported significantly lower urinary pesticide levels among children consuming organic versus conventional diets, but studies of biomarker and nutrient levels in serum, urine, breast milk, and semen in adults did not identify clinically meaningful differences. All estimates of differences in nutrient and contaminant levels in foods were highly heterogeneous except for the estimate for phosphorus; phosphorus levels were significantly higher than in conventional produce, although this difference is not clinically significant. The risk for contamination with detectable pesticide residues was lower among organic than conventional produce (risk difference, 30% [CI, -37% to -23%]), but differences in risk for exceeding maximum allowed limits were small. Escherichia coli contamination risk did not differ between organic and conventional produce. Bacterial contamination of retail chicken and pork was common but unrelated to farming method. However, the risk for isolating bacteria resistant to 3 or more antibiotics was higher in conventional than in organic chicken and pork (risk difference, 33% [CI, 21% to 45%]).
Limitation: Studies were heterogeneous and limited in number, and publication bias may be present.
Conclusion: The published literature lacks strong evidence that organic foods are significantly more nutritious than conventional foods. Consumption of organic foods may reduce exposure to pesticide residues and antibiotic-resistant bacteria.
I don’t really have a problem with the description of their methods. Again, I hope to return to this and talk in more detail.
Pic Source: http://goo.gl/LQRzE
September 23, 2012
Looking forward to learning more Chad Haney and thanks.
September 25, 2012
Thanks for posting this! I hope that we can get beyond the headline grabbing aspects of many recent reports and create situations in which we can have deeper and more serious discussions regarding the science of such things as healthy and sustainable agriculture. And the Science Sunday SSHOw is off to a good start on aiding this. (Even though I was only able to view this after the fact).
September 25, 2012
Thank Gaythia Weis I apologize that I did not have time to dig into this deeper.
October 23, 2012
Just to be clear — does the “publication bias” to which you refer encompass the “Failure(s) to Replicate” which were never published? Because those studies and their non-significant results would, in my judgment, affect any probabilities concocted from any meta-analysis.
October 23, 2012
William McGarvey I’m trying to figure out what point you are driving toward. Research is always time and money limited and thus oriented to favor studies that are likely to yield, in the opinion of the researcher or Principal Investigator, the biggest “bang for the buck”. Research publications do not give an unbiased sampling of the previously unknown universe, even on a small topic. Thus a meta analysis sampling would need to be weighted and analyzed in ways that take those sorts of biases into account. But in looking at published data, of course non published data is not included.
October 23, 2012
Gaythia Weis I concur with much of what you’ve written. I’m pointing, though, to (with apologies to Donald Rumsfeld) the “unknown unknowns”. I would hope that several “corrections” might be applied in any such meta-analysis. All I’m asserting, however, is that the “true” probabilities in their results may remain difficult to determine, and that meta-analysis is but a step beyond literature review. To give it more credence than that is to be at risk for the fallacy of false precision.
October 23, 2012
I agree Bill. Meta-analysis does have its flaws and that’s one of them.
October 23, 2012
William McGarvey one of the things that a good Meta Analsis could reveal, IMHO would be some of the possible “unknown unknowns”. So we are, I believe, largely in agreement. As I tried to explain in the series of posts I did that originally inspired this interaction with Chad Haney, regarding the Stanford based scientists organic foods study: https://plus.google.com/u/0/100946817944952712900/posts/RUMc6sAP3Tu
I attempted to emphasize that science is an ongoing process and Meta Analysis is just one step in that process:
“Meta-analysis and related statistical tools can be powerful as part of a systematic review of a field of study. These sorts of studies can be of great aid in identifying patterns and in determining gaps in what is known. “
and gave my opinion that:
_”I personally find this Stanford report to be accurate as written. My concerns lie more with what I see as the headline grabbing nature of it’s presentation to the public.
If it is used to inspire further, more carefully controlled [experimental] work, that would be a good thing. If it is used to more carefully define how we determine “organic” labeling, that would also be excellent. If it is simply used to cast doubt, I think that would be a negative”_