Bad Science
Sep. 6th, 2010 08:14 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
purplecatWe saw Ben Goldacre talk at Eastercon though, I think, I already had his book, Bad Science, following a recommendation from ![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
lil_shepherd. It is more carefully structured than his talk was and has more space to explore its argument, which is a definite plus because he never quite got to his point at Eastercon.
In general the book walks through a progression of bogus science starting with the obvious (the example used here are detox products which behave the same way whether in contact with a human or not) and moving through ever more sophisticated forms of bad science, from trials which are uncontrolled, trials which are not blinded to trials which perform essentially unfair comparisons. In the course of this there is a completely fascinating chapter on the placebo effect and a look at experiments that have shown that some placebos are more effective than others.
A running theme throughout the book is media complicity in the promotion of bad science. I know (at least one) of the journalistic friends of my acquaintance feels Goldacre is being unfair to the media in his arguments and painting them as straightforward villains when the reality, in his own phrase, is a little bit more complicated than that. I think actually Goldacre's essential argument here is firstly that evidence-based medicine is woefully underrepresented in our education system, given that the vast majority of science stories in the media are about health. He couples this with an observation that most journalists, and in particular generalist journalists (as opposed to science correspondants) are arts graduates and tend to view science as an argument between opposing authority figures with the evidence as a sideline. Therefore the accuracy of a science story depends on the reputation of the scientist and not on the actual evidence presented. It is certainly pretty damning that so little of the reporting on the MMR health scare went anywhere near a paper's science correspondant. Goldacre makes the point that, despite the media's current interest in presenting Andrew Wakefield as an out-and-out villain, MMR isn't safe because Andrew Wakefield was unethical, it is safe because all the evidence points to it being safe. Andrew Wakefield's character should be irrelevant - although the arena became complicated later, it should have been obvious to anyone with a basic grounding in scientific trials that he was making vastly inflated claims for a study involving only twelve, self-selected subjects.
The book itself stands primarily as a call to teach the basics of evidence-based medicine in schools: what constitutes a trial, what constitutes a trial against control, why trials should be blinded, the problems with applying statistics post-facto, and so on. Goldacre returns repeatedly throughout to the theme of checking the papers for yourself and an insistence that, by and large, these are easy to understand with only a few basics in experimental design and analysis. It is unfortunate therefore, that in some of the later chapters where he is looking at more sophisticated forms of bad science he does actually have to tangle with some fairly complicated subject matter. Early on he makes the point that you should read the evidence in a scientific paper first and only then read the conclusions since these are often written to play up or down certain results. However he then berates the opponents of MMR for ignoring the conclusions in a Cochrane review* and instead focusing on the review's internal criticisms of the various trials it compares.
I didn't always agree with Goldacre and I think he could be accused of simplifying issues in the face of his own assertion that things are nearly always a little bit more complicated than they seem. But the book is immensely readable and I can't disagree with his basic thesis that we should invest more effort into education on the basics of evidence-based medicine especially in face of the media and public obsession with health scares and miracle cures. As primer in how to spot bad science in the medical field, you could do a lot worse than starting here.
Incidentally (though I'm here using Goldacre as an authority figure), three facts jumped out at me from the book. Firstly Vitamin C has been shown to have no preventative effect on the common cold (Vitamin C has always been my least favourite vitamin, possibly because I dislike oranges, so I felt a smug satisfaction at this news). Secondly I've been slowly reading my way through McGee's On Food and Cooking - one of the molecular gastronomy bibles. McGee is big on antioxidants, Goldacre is not. This casts something of a shadow over the whole molecular gastronomy program since it would seem that their scientific approach to cookery is as vulnerable to the lure of quackery as wider journalism. Lastly Goldacre mentions, in his chapter on MMR, that there was a similarly unfounded Whooping Cough vaccine scare in the 1970s. I can actually remember suffering through Whooping Cough and being told I hadn't been vaccinated against it because of the risk of brain damage.
*Cochrane reviews perform meta-analyses, i.e. they analyse the results of all available trials on some treatment.claraste who does systematic reviews of medical literature professionally was rather aghast when I told her Goldacre was implying that lay people should read Cochrane reviews since they are (so she tells me) written to a very rigid template and assume a lot of technical knowledge on the construction of meta-analyses. It is for this very reason that Goldacre insists more attention should have been paid to the conclusion, but at this point you've reached a level of complexity (when should you give more weight to paper's conclusions and when to what you can understand of the evidence presented) that, I suspect, will cause a lot of people to retreat into "I just don't understand" territory.
This entry was originally posted at http://purplecat.dreamwidth.org/17637.html.
lil_shepherd. It is more carefully structured than his talk was and has more space to explore its argument, which is a definite plus because he never quite got to his point at Eastercon.In general the book walks through a progression of bogus science starting with the obvious (the example used here are detox products which behave the same way whether in contact with a human or not) and moving through ever more sophisticated forms of bad science, from trials which are uncontrolled, trials which are not blinded to trials which perform essentially unfair comparisons. In the course of this there is a completely fascinating chapter on the placebo effect and a look at experiments that have shown that some placebos are more effective than others.
A running theme throughout the book is media complicity in the promotion of bad science. I know (at least one) of the journalistic friends of my acquaintance feels Goldacre is being unfair to the media in his arguments and painting them as straightforward villains when the reality, in his own phrase, is a little bit more complicated than that. I think actually Goldacre's essential argument here is firstly that evidence-based medicine is woefully underrepresented in our education system, given that the vast majority of science stories in the media are about health. He couples this with an observation that most journalists, and in particular generalist journalists (as opposed to science correspondants) are arts graduates and tend to view science as an argument between opposing authority figures with the evidence as a sideline. Therefore the accuracy of a science story depends on the reputation of the scientist and not on the actual evidence presented. It is certainly pretty damning that so little of the reporting on the MMR health scare went anywhere near a paper's science correspondant. Goldacre makes the point that, despite the media's current interest in presenting Andrew Wakefield as an out-and-out villain, MMR isn't safe because Andrew Wakefield was unethical, it is safe because all the evidence points to it being safe. Andrew Wakefield's character should be irrelevant - although the arena became complicated later, it should have been obvious to anyone with a basic grounding in scientific trials that he was making vastly inflated claims for a study involving only twelve, self-selected subjects.
The book itself stands primarily as a call to teach the basics of evidence-based medicine in schools: what constitutes a trial, what constitutes a trial against control, why trials should be blinded, the problems with applying statistics post-facto, and so on. Goldacre returns repeatedly throughout to the theme of checking the papers for yourself and an insistence that, by and large, these are easy to understand with only a few basics in experimental design and analysis. It is unfortunate therefore, that in some of the later chapters where he is looking at more sophisticated forms of bad science he does actually have to tangle with some fairly complicated subject matter. Early on he makes the point that you should read the evidence in a scientific paper first and only then read the conclusions since these are often written to play up or down certain results. However he then berates the opponents of MMR for ignoring the conclusions in a Cochrane review* and instead focusing on the review's internal criticisms of the various trials it compares.
I didn't always agree with Goldacre and I think he could be accused of simplifying issues in the face of his own assertion that things are nearly always a little bit more complicated than they seem. But the book is immensely readable and I can't disagree with his basic thesis that we should invest more effort into education on the basics of evidence-based medicine especially in face of the media and public obsession with health scares and miracle cures. As primer in how to spot bad science in the medical field, you could do a lot worse than starting here.
Incidentally (though I'm here using Goldacre as an authority figure), three facts jumped out at me from the book. Firstly Vitamin C has been shown to have no preventative effect on the common cold (Vitamin C has always been my least favourite vitamin, possibly because I dislike oranges, so I felt a smug satisfaction at this news). Secondly I've been slowly reading my way through McGee's On Food and Cooking - one of the molecular gastronomy bibles. McGee is big on antioxidants, Goldacre is not. This casts something of a shadow over the whole molecular gastronomy program since it would seem that their scientific approach to cookery is as vulnerable to the lure of quackery as wider journalism. Lastly Goldacre mentions, in his chapter on MMR, that there was a similarly unfounded Whooping Cough vaccine scare in the 1970s. I can actually remember suffering through Whooping Cough and being told I hadn't been vaccinated against it because of the risk of brain damage.
*Cochrane reviews perform meta-analyses, i.e. they analyse the results of all available trials on some treatment.
claraste who does systematic reviews of medical literature professionally was rather aghast when I told her Goldacre was implying that lay people should read Cochrane reviews since they are (so she tells me) written to a very rigid template and assume a lot of technical knowledge on the construction of meta-analyses. It is for this very reason that Goldacre insists more attention should have been paid to the conclusion, but at this point you've reached a level of complexity (when should you give more weight to paper's conclusions and when to what you can understand of the evidence presented) that, I suspect, will cause a lot of people to retreat into "I just don't understand" territory.This entry was originally posted at http://purplecat.dreamwidth.org/17637.html.
(no subject)
Date: 2010-09-06 09:08 pm (UTC)I did feel the book skirted a fine line between being readable and over-indulging in polemic. I suspect that with more exposure to his writing I might have stronger opinions about which side of the line it fell.
Bad economics would be great. I've had people try to explain short-selling and naked short-selling to me several times. I think I've grasped how one of them works, but I'm not sure which. As for the rest of economics - I think I've just about got a grip on how supply and demand interact in general terms....