Bad science → bad headlines

Bad science → bad headlines

Today’s #SSHOw (http://goo.gl/0eGrh) will be discussing GMO corn and in particular a poorly done experiment/publication that sparked the media storm (Séralini et al Food and Toxicology 2012).  http://goo.gl/5GOWa

Orac dissects the paper quite nicely, although I think he repeatedly says mice when he means rats.  http://goo.gl/SSE2F

The two areas I will comment on are the tumor rat model and a statistical issue dealing with multiple comparisons.

Spontaneous Tumors in Rats

Orac points to a study in 1979 where 81% of the Sprague-Dawley rats, the same strain used in the Séralini paper, develop tumors. When you use an animal model looking at tumor development, you need to know the prevalence of spontaneous tumor development. The control group(s) have to be designed such that you can differentiate between “normal” spontaneous tumor development in the control groups vs. the experimental groups. Part of that design is having sufficient number of animals to have statistical power. Using previously published data, the authors could have done a power analysis to determine the proper sample size. For these types of studies, where you are not doing intricate daily or weekly interventions/experiments, i.e., just keeping the animals for long periods while looking for mortality, it is not uncommon to have 3-5 times the number of animals used in the Séralini study.

As an example, I had the privilege to collaborate with Prof Morris Pollard at Notre Dame who developed the Lobund-Wistar rat model. Lobund-Wistar rats spontaneously develop prostate adenocarcinoma (PA) at a mean age of 26 months. In the publication of the model, out of 72 L-W rats, 19 (26%) developed large PAs. Imagine if Prof Pollard only used 10 male rats as in the Séralini study. 

The Lobund-Wistar rat model of prostate cancer.

Pollard M.

J Cell Biochem Suppl. 1992;16H:84-8.

http://www.ncbi.nlm.nih.gov/pubmed/1289678

We use a transgenic mouse model for spontaneous ductal carcinoma in situ. Invasive carcinomas develop in 100% of the mice. The mice are called SV40-Tag mice based on the C3(1)/Tag mice. SV40-Tag stands for Simian virus 40 T-antigen, a trans-activating protein, which are essential for viral gene expression.

Green et al Oncogene 2000 

http://www.ncbi.nlm.nih.gov/pubmed/10713685

The point is you have to know the tumor prevalence in the rodent model you are using and plan the control groups accordingly.

Multiple Comparisons/Sample Size

The study mentions that they use Discriminant Analysis (DA) to partition groups, i.e., you lump all the variables (factors) together and use DA to flesh out which factors influence the outcome, e.g., tumor size, biochemical markers, etc. In image analysis we use Linear Discriminant Analysis (LDA,http://en.wikipedia.org/wiki/Discriminant_analysis,  http://goo.gl/oyNzh) to segment (classify) pixels. Say you want to automatically segment tumor from normal tissue using several image types of the same sample. You have thousands of pixels to work with, not 10. The method isn’t robust with 10 or less samples in the 20 groups used (note I separated the male and female groups in the Séralini study). Also, in the context of machine learning, you have to have a training set. In my example you give the program a set of pixels that you know belong to each group before testing the pixels you want to classify.

A quick review on null hypothesis testing. A type I error is when the null hypothesis is true but is rejected, i.e., a false positive. A type II error is when the null hypothesis is false but is incorrectly accepted as true, a false negative. Remember the null hypothesis can never be proven.

Here are examples from the Wiki:

Suppose the treatment is a new way of teaching writing to students, and the control is the standard way of teaching writing. Students in the two groups can be compared in terms of grammar, spelling, organization, content, and so on. As more attributes are compared, it becomes more likely that the treatment and control groups will appear to differ on at least one attribute by random chance alone.

Suppose we consider the efficacy of a drug in terms of the reduction of any one of a number of disease symptoms. As more symptoms are considered, it becomes more likely that the drug will appear to be an improvement over existing drugs in terms of at least one symptom.

Suppose we consider the safety of a drug in terms of the occurrences of different types of side effects. As more types of side effects are considered, it becomes more likely that the new drug will appear to be less safe than existing drugs in terms of at least one side effect.

http://en.wikipedia.org/wiki/Multiple_comparisons

Statistical power is the probability of committing a type II error, false negative. Prof Pollard’s study has a statistical power of around 97% while the Séralini study is probably closer to 45%.

In Memoriam

I the process of digging up the study by Prof Pollard, I realized he had passed away. I met him when he was in his late 80s, to do an experiment for him at the University of Chicago. He was an impressive man and scientist. It is really a shame he is most often known for his son.

http://goo.gl/DdQP8

Pollard worked at all of these things until his very last days. “I can’t imagine doing anything else,” he said recently. “I think if you are doing something meaningful and important and you stop doing it, you’ll always look back with regret.”

L-W Rat pic: http://goo.gl/hzP6P

For #ScienceSunday  curated by Allison Sekuler Rajini Rao Robby Bowles and me.