![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Entry tags:
Bad Science
We saw Ben Goldacre talk at Eastercon though, I think, I already had his book, Bad Science, following a recommendation from ![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
lil_shepherd. It is more carefully structured than his talk was and has more space to explore its argument, which is a definite plus because he never quite got to his point at Eastercon.
In general the book walks through a progression of bogus science starting with the obvious (the example used here are detox products which behave the same way whether in contact with a human or not) and moving through ever more sophisticated forms of bad science, from trials which are uncontrolled, trials which are not blinded to trials which perform essentially unfair comparisons. In the course of this there is a completely fascinating chapter on the placebo effect and a look at experiments that have shown that some placebos are more effective than others.
A running theme throughout the book is media complicity in the promotion of bad science. I know (at least one) of the journalistic friends of my acquaintance feels Goldacre is being unfair to the media in his arguments and painting them as straightforward villains when the reality, in his own phrase, is a little bit more complicated than that. I think actually Goldacre's essential argument here is firstly that evidence-based medicine is woefully underrepresented in our education system, given that the vast majority of science stories in the media are about health. He couples this with an observation that most journalists, and in particular generalist journalists (as opposed to science correspondants) are arts graduates and tend to view science as an argument between opposing authority figures with the evidence as a sideline. Therefore the accuracy of a science story depends on the reputation of the scientist and not on the actual evidence presented. It is certainly pretty damning that so little of the reporting on the MMR health scare went anywhere near a paper's science correspondant. Goldacre makes the point that, despite the media's current interest in presenting Andrew Wakefield as an out-and-out villain, MMR isn't safe because Andrew Wakefield was unethical, it is safe because all the evidence points to it being safe. Andrew Wakefield's character should be irrelevant - although the arena became complicated later, it should have been obvious to anyone with a basic grounding in scientific trials that he was making vastly inflated claims for a study involving only twelve, self-selected subjects.
The book itself stands primarily as a call to teach the basics of evidence-based medicine in schools: what constitutes a trial, what constitutes a trial against control, why trials should be blinded, the problems with applying statistics post-facto, and so on. Goldacre returns repeatedly throughout to the theme of checking the papers for yourself and an insistence that, by and large, these are easy to understand with only a few basics in experimental design and analysis. It is unfortunate therefore, that in some of the later chapters where he is looking at more sophisticated forms of bad science he does actually have to tangle with some fairly complicated subject matter. Early on he makes the point that you should read the evidence in a scientific paper first and only then read the conclusions since these are often written to play up or down certain results. However he then berates the opponents of MMR for ignoring the conclusions in a Cochrane review* and instead focusing on the review's internal criticisms of the various trials it compares.
I didn't always agree with Goldacre and I think he could be accused of simplifying issues in the face of his own assertion that things are nearly always a little bit more complicated than they seem. But the book is immensely readable and I can't disagree with his basic thesis that we should invest more effort into education on the basics of evidence-based medicine especially in face of the media and public obsession with health scares and miracle cures. As primer in how to spot bad science in the medical field, you could do a lot worse than starting here.
Incidentally (though I'm here using Goldacre as an authority figure), three facts jumped out at me from the book. Firstly Vitamin C has been shown to have no preventative effect on the common cold (Vitamin C has always been my least favourite vitamin, possibly because I dislike oranges, so I felt a smug satisfaction at this news). Secondly I've been slowly reading my way through McGee's On Food and Cooking - one of the molecular gastronomy bibles. McGee is big on antioxidants, Goldacre is not. This casts something of a shadow over the whole molecular gastronomy program since it would seem that their scientific approach to cookery is as vulnerable to the lure of quackery as wider journalism. Lastly Goldacre mentions, in his chapter on MMR, that there was a similarly unfounded Whooping Cough vaccine scare in the 1970s. I can actually remember suffering through Whooping Cough and being told I hadn't been vaccinated against it because of the risk of brain damage.
*Cochrane reviews perform meta-analyses, i.e. they analyse the results of all available trials on some treatment.claraste who does systematic reviews of medical literature professionally was rather aghast when I told her Goldacre was implying that lay people should read Cochrane reviews since they are (so she tells me) written to a very rigid template and assume a lot of technical knowledge on the construction of meta-analyses. It is for this very reason that Goldacre insists more attention should have been paid to the conclusion, but at this point you've reached a level of complexity (when should you give more weight to paper's conclusions and when to what you can understand of the evidence presented) that, I suspect, will cause a lot of people to retreat into "I just don't understand" territory.
This entry was originally posted at http://purplecat.dreamwidth.org/17637.html.
lil_shepherd. It is more carefully structured than his talk was and has more space to explore its argument, which is a definite plus because he never quite got to his point at Eastercon.In general the book walks through a progression of bogus science starting with the obvious (the example used here are detox products which behave the same way whether in contact with a human or not) and moving through ever more sophisticated forms of bad science, from trials which are uncontrolled, trials which are not blinded to trials which perform essentially unfair comparisons. In the course of this there is a completely fascinating chapter on the placebo effect and a look at experiments that have shown that some placebos are more effective than others.
A running theme throughout the book is media complicity in the promotion of bad science. I know (at least one) of the journalistic friends of my acquaintance feels Goldacre is being unfair to the media in his arguments and painting them as straightforward villains when the reality, in his own phrase, is a little bit more complicated than that. I think actually Goldacre's essential argument here is firstly that evidence-based medicine is woefully underrepresented in our education system, given that the vast majority of science stories in the media are about health. He couples this with an observation that most journalists, and in particular generalist journalists (as opposed to science correspondants) are arts graduates and tend to view science as an argument between opposing authority figures with the evidence as a sideline. Therefore the accuracy of a science story depends on the reputation of the scientist and not on the actual evidence presented. It is certainly pretty damning that so little of the reporting on the MMR health scare went anywhere near a paper's science correspondant. Goldacre makes the point that, despite the media's current interest in presenting Andrew Wakefield as an out-and-out villain, MMR isn't safe because Andrew Wakefield was unethical, it is safe because all the evidence points to it being safe. Andrew Wakefield's character should be irrelevant - although the arena became complicated later, it should have been obvious to anyone with a basic grounding in scientific trials that he was making vastly inflated claims for a study involving only twelve, self-selected subjects.
The book itself stands primarily as a call to teach the basics of evidence-based medicine in schools: what constitutes a trial, what constitutes a trial against control, why trials should be blinded, the problems with applying statistics post-facto, and so on. Goldacre returns repeatedly throughout to the theme of checking the papers for yourself and an insistence that, by and large, these are easy to understand with only a few basics in experimental design and analysis. It is unfortunate therefore, that in some of the later chapters where he is looking at more sophisticated forms of bad science he does actually have to tangle with some fairly complicated subject matter. Early on he makes the point that you should read the evidence in a scientific paper first and only then read the conclusions since these are often written to play up or down certain results. However he then berates the opponents of MMR for ignoring the conclusions in a Cochrane review* and instead focusing on the review's internal criticisms of the various trials it compares.
I didn't always agree with Goldacre and I think he could be accused of simplifying issues in the face of his own assertion that things are nearly always a little bit more complicated than they seem. But the book is immensely readable and I can't disagree with his basic thesis that we should invest more effort into education on the basics of evidence-based medicine especially in face of the media and public obsession with health scares and miracle cures. As primer in how to spot bad science in the medical field, you could do a lot worse than starting here.
Incidentally (though I'm here using Goldacre as an authority figure), three facts jumped out at me from the book. Firstly Vitamin C has been shown to have no preventative effect on the common cold (Vitamin C has always been my least favourite vitamin, possibly because I dislike oranges, so I felt a smug satisfaction at this news). Secondly I've been slowly reading my way through McGee's On Food and Cooking - one of the molecular gastronomy bibles. McGee is big on antioxidants, Goldacre is not. This casts something of a shadow over the whole molecular gastronomy program since it would seem that their scientific approach to cookery is as vulnerable to the lure of quackery as wider journalism. Lastly Goldacre mentions, in his chapter on MMR, that there was a similarly unfounded Whooping Cough vaccine scare in the 1970s. I can actually remember suffering through Whooping Cough and being told I hadn't been vaccinated against it because of the risk of brain damage.
*Cochrane reviews perform meta-analyses, i.e. they analyse the results of all available trials on some treatment.
claraste who does systematic reviews of medical literature professionally was rather aghast when I told her Goldacre was implying that lay people should read Cochrane reviews since they are (so she tells me) written to a very rigid template and assume a lot of technical knowledge on the construction of meta-analyses. It is for this very reason that Goldacre insists more attention should have been paid to the conclusion, but at this point you've reached a level of complexity (when should you give more weight to paper's conclusions and when to what you can understand of the evidence presented) that, I suspect, will cause a lot of people to retreat into "I just don't understand" territory.This entry was originally posted at http://purplecat.dreamwidth.org/17637.html.
no subject
But then at the same time, I come back to something a friend of mine from Magdalen said: Ben Goldacre is one of three people (the other two being Stephen Fry and Brian Cox) that people follow on Twitter to 'prove' they are clever. And if you ever read the comments on his blog, you'll see that there is some truth in this.
Mind you, I'd love someone to do the same thing for economics - a 'Bad Economics' if you will. Freakonomics and Superfreakonomics go in that direction, but are more informative than argumentative, and only cover micro-, never macroeconomics.
no subject
http://tardis.wikia.com/wiki/Scientific_Reform_Society
no subject
I did feel the book skirted a fine line between being readable and over-indulging in polemic. I suspect that with more exposure to his writing I might have stronger opinions about which side of the line it fell.
Bad economics would be great. I've had people try to explain short-selling and naked short-selling to me several times. I think I've grasped how one of them works, but I'm not sure which. As for the rest of economics - I think I've just about got a grip on how supply and demand interact in general terms....
no subject
The other day there was some news story that mentioned him in a manner he thought incorrect, so he posted a comment correction: when the comment had not been published only a few hours later he threw a hissy fit on his blog. Unimpressive in a man that wants people to check references and validate their views, I thought (particularly as the comment was eventually published in what seemed like an entirely reasonable timescale...)
There are some podcasts somewhere of his stuff on placebos they are quite interesting.
no subject
no subject
The issue of science graduates not going into journalism is one that does need highlighting. Sure, there are some who want to cover health, but that's not the same thing at all -- they veer towards the health and beauty field.
no subject
It wouldn't avoid the sneakier bad science creeping in, but would hopefully stop the whole nation getting its knickers in a twist over a study involving 12 subjects.
*from a certain point of view anyway.
no subject
no subject
Similarly a better grounding would allow them to ask more searching questions. I don't know if you came across the Toads leaving Aquila story (some toads that were being studied near Aqulia vanished in the days preceding the Earthquake). It was mentioned on the Today programme and John Humphrys is quite intelligent enough and sufficiently unafraid of putting interviewees on the spot, to make the points that correlation does not imply causation, and that you are on dodgy ground drawing strong conclusions from noticing something after the fact that you weren't initially looking for - fairly basic points which wider basic scientific literacy would sort of require a journalist to make (if that makes sense). In this case the scientist involved has done nothing more than say that there needs to be more work, I hasten to add, which is precisely what Wakefield should have done, instead he chose to make medical recommendations for alternative forms of vaccine and so on and so forth.
I think also, since at some level we always have to invoke authority figures, I'd expect journalists to shine the same sort of light onto the false claims, and networks of complicity that can exist in "science" as they do to politicians. Again wider scientific literacy would have made a paper blush to claim that a man who had gained a PhD via a correspondance course with one of America's dodgier universities and who was operating out of his garden shed was the UK's foremost expert on MRSA. As it stands that kind of understanding that not all PhD's are equal and not all "labs" (to be generous) work to the same standards is sufficiently lacking in the public consciousness that it doesn't appear to occur to journalists to check that kind of thing.
no subject
Neuromancer (UK's foremost expert on fiscal policy. No, honestly.)
no subject
You can write some articles for my economics blog then!
no subject
There's clearly a class of paper which has no real interest in the truth and that's a much wider problem than in the public understanding of sicence, but I like to think there are a lot that still do and these are making mistakes which you'd hope were easily avoidable, though I seem to have picked the wrong example here.
no subject
no subject
no subject
no subject
Neuromancer
no subject
no subject
no subject
As such I think he is at times, as you say, overly harsh on journalists. By the same count, BG obviously has a good point regarding science education and not leaving science reporting in the hands of generalist journalists with humanities degrees who, however intelligent they are, have no grounding in the technical detail of the subject.
There is probably also a tangential point to be made about the value of a baccalaureate type qualification at 18 which would require a combination of both arts and sciences. GCSE-level science and maths are still fairly basic (and, if I recall, often "wrong" or at least very over-simplified), but are the point at which many highly-educated and intelligent people stop their studying of such subjects.
no subject
no subject
I would like to see more people know about economics (part of the reason for my recent economics posts) and that is something that could be compatible with teaching younger children about scientific methodology, bias, scepticism, peer review etc. First year of secondary school would be a good time to run lessons on 'How to think' that could cover all of these areas.
no subject
There is also some kind of critical thought A level these days, but when I was an Admissions Officer I was told to ignore both it and General Studies.