Pope of debunkers February 15, 2009Posted by dissident93 in Iraq mortality.
Tim Lambert runs a “science blog” which at times successfully debunks rightwing punditry. At other times, Lambert seems out of his depth. For example, his attempt at “analysis” of media inaccuracy (over Iraq Body Count’s figures) was riddled with errors. He grudgingly corrected this after I pointed out the mistakes, but only partially (it still contains obvious howlers). Remarkably, Lambert admitted that he hadn’t read to the end of the media articles that he was supposedly analyzing.
More recently, Lambert seemed out of his depth when attempting to debunk an award-winning paper published by the Journal of Peace Research (Bias in Epidemiological Studies of Conflict Mortality, by Neil F. Johnson et al). This paper is better known as the “main street bias” work which was critical of the 2006 Lancet study on Iraqi deaths (and which has been expanded in a new paper for the European Physics Letters journal).
Lambert’s attempts to debunk this research (which he’d previously dismissed as “bogus”) boil down to three banal observations:
- That it’s possible to disagree with the example parameter values used in the paper (values used to suggest possible bias in the Lancet Iraq study – nobody knows the actual values, partly because the Lancet authors won’t release necessary information). (Main blog post)
- That you can construct a hypothetical case which ignores the premises of the main street bias model in order to “demonstrate” that you can ignore those premises. (Comment #55)
- That you can falsely impute assumptions to the MSB paper if you can’t come up with anything better to debunk it with. (Comment #144)
Lambert wouldn’t see it that way, of course, but his attempts to debunk the study are themselves pretty convincingly debunked (eg see comments #111, #153, #180, and my own #3, #132, #150, etc) after which he doesn’t have much more to say. Still, his blog is sometimes worth reading – for the scrutiny of spurious punditry, and for the hilarious outbursts of pompous credentialism (such as when a poster who called himself Expert Statistician wrote: “Go get a PhD in stats, then we’ll talk”).
Postscript – Lambert on Main Street Bias
[The following was originally a separate post, from 22/2/09 – I’ve incorporated it into the above to remove clutter]
Tim Lambert wrote a second piece attempting to debunk MSB. This time he drew a corner of a map, hoping to show that there may be no sampling bias from “main” streets in the 2006 Lancet Iraq study.
I’ve drawn a map (see below) illustrating the problems with Lambert’s assumptions, although you probably need to be familiar with this debate for it to make any sense.
Please click on the map to see it at readable size.
Lambert has now replied to me that the road he redesignated as a “main” street (but which wasn’t shown in his own map) was the one which runs upwards through junction A (on my map). In his blog, Lambert describes this as an “obvious” main street, but it’s nothing of the sort (it’s half the width of Lambert’s other designated “main” street, for example).
The best that Lambert can claim for one of his secondary (or “cross”) roads (road 1 in my map) is that it joins another cross road (2) at junction A, and that the road which he has arbitrarily designated as a “main” street runs through the same junction.
In other words, even if a survey team agreed with Lambert’s arbitrary designation of the junction A road as a “main” street, it’s debatable whether (using the published Lancet sampling methodology) they would select road 1 as a cross street. Looking at Junction A, it seems equally (or perhaps more) likely that they’d class road 2 as the secondary street, with road 1 as a tertiary road leading off road 2. But nobody actually knows, since nobody knows how the Lancet sampling scheme worked in reality.
What this ambiguity over classes of road shows, at this level of detail, is that Lambert is misleading his readers when he claims that the MSB map (which he redrew) is “wrong”. The most he can say is that he has a different subjective designation of roads, which has its own problems in terms of plausibility.
Lambert’s map left out the least plausible part of his selection scheme.
Violating “fundamental standards of science”? February 4, 2009Posted by dissident93 in Iraq mortality.
In what ABC News calls a “highly unusual rebuke”, the American Association for Public Opinion Research (AAPOR) has accused Gilbert Burnham, lead author of the 2006 Lancet study on Iraqi deaths, of violating its code of professional ethics.
According to ABC, the Association last brought such a charge of ethics violation 12 years ago, against rightwing pollster Frank Luntz. Burnham is accused of repeatedly refusing “to make public essential facts about his research” (on Iraqi deaths). AAPOR holds that researchers must make available for public disclosure essential information such as the wording of survey questions and other basic methodological details.
Burnham isn’t a member of AAPOR, but that isn’t particularly relevant given that the charge is of a serious breach of widely accepted professional ethics. In fact AAPOR seem to go out of their way to stress the gravity of the case. In their press release, AAPOR’s President, Richard Kulka, says the following:
“When researchers draw important conclusions and make public statements and arguments based on survey research data, then subsequently refuse to answer even basic questions about how their research was conducted, this violates the fundamental standards of science, seriously undermines open public debate on critical issues, and undermines the credibility of all survey and public opinion research. These concerns have been at the foundation of AAPOR’s standards and professional code throughout our history, and when these principles have clearly been violated, making the public aware of these violations is in integral part of our mission and values as a professional organization.” [My emphasis – RS]
David Marker (chair of the American Statistical Association’s Scientific and Public Affairs Advisory Committee), in a Methodological Review of the Lancet study, raises the issue of the AAPOR-published “Best Methods” for prevention of interviewer falsification in survey research. He comments:
A few years ago, 35 leading survey researchers issued a consensus statement on how to minimize interviewer falsification of data (AAPOR 2003). This statement has been endorsed by the American Association for Public Opinion Research and the Survey Research Methods Section of the American Statistical Association. They listed eight factors that could affect falsification rates. Inadequate supervision, poor quality control and off-site isolation of interviewers were three of those factors that are present in this [Lancet] study. The remaining five factors (training on falsification, interviewer motivation, inadequate compensation, piece-rate compensation, and excessive workload) are harder to assess in this situation due to the limited information available on these topics.
In a paper to be published in Defence and Peace Economics, Professor Michael Spagat, of Royal Holloway, had previously documented areas in which the Lancet study appears to violate AAPOR’s code of professional ethics and practices: Ethical and Data-Integrity Problems in the Second Lancet Survey of Mortality in Iraq