Project Censored correspondence July 1, 2010Posted by dissident93 in Project Censored.
While archiving emails I came across the eye-opening correspondence I had with Project Censored, which led to this post. I’d complained that their “Over One Million Iraqi Deaths” story effectively “censored” all the research (including major studies such as WHO/IFHS) which refuted their headline.
Their (then) director, Peter Phillips, replied to me at length, together with his researchers, Michael Schwartz & Joshua Holland. But their replies were riddled with errors (see list below). And nearly all of these errors were second-hand – they’d been circulating on the web for some time, and most had already been refuted. This indicated to me a very poor level of research – worse, ironically, than many pieces I’d seen in the mainstream media (which at least acknowledged the existence of studies such as WHO/IFHS).
I took the time to reply in detail, correcting the errors, etc (see list below). Their responses to this were interesting. Joshua Holland wrote that he was too busy to respond as he was covering the election for a few weeks (I never heard from him again). Michael Schwartz replied (abruptly) that he wasn’t interested in “recapitulating these disputes” (even though it was he who’d raised these detailed disputes, as justification for airbrushing IFHS, etc, out of the picture – I hadn’t mentioned them at all). I never heard anything further from him.
Peter Phillips eventually replied again: “I have given your comments serious thought and have decide not to change the title on our headline. I believe that over one million Iraqis have been died because of the US invasion and occupation and there is significant evidence to support that conclusion.” (15/10/2008).
In other words, a complete avoidance of the issue I’d raised: that regardless of his own belief, there was no scientific consensus over the “one million” figure (many authorities in the field did not accept this figure, as I documented in my original piece), and that he had effectively “censored” a large body of research because it didn’t support his “belief”.
Here’s the list of corrections to the errors of Project Censored’s researchers (which I sent to them on 30/9/08):
ERRORS & DETAILS
1. Joshua Holland writes that “IFHS also looked only at civilians”. This is incorrect. IFHS (like Lancet 2006) includes “combatants” as well as “civilians” in its estimate of violent deaths. (Ref: IFHS Q&A, p5: http://tinyurl.com/4o3w82)
2. Joshua Holland writes that “IFHS also omitted 11% of its sample”. This is correct, but Lancet 2006 also omitted some of its planned clusters (6%). IFHS made an effort to compensate for the omitted clusters; Lancet 2006 did not. Furthermore, IFHS made an effort to reflect regional population changes from migration during the survey period. Lancet 2006 did not.
3. Michael Schwartz writes that “the IFHS methodology has resulted in underestimates of both the number of total excess deaths and total violent deaths”. This is clearly incorrect, as IFHS provides no estimate for “total excess deaths” (IFHS provides an estimate for total violent deaths only). I’d be grateful if Michael could indicate which estimate of “total excess deaths” he is referring to here. (Perhaps he means the widely circulated estimate of 400,000 which has been falsely attributed to IFHS? More on this below). [*See July 2010 footnote on the "400,000" figure]
Michael claims that IFHS “underestimates” violent deaths – an unsupported assertion. Interestingly, the IFHS paper actually makes the argument that conflict surveys tend to underestimate violent deaths (mainly due to survivor bias), and as a result IFHS have adjusted their estimate upwards by about 54% to account for this proposed bias. Lancet 2006 doesn’t make this upward adjustment.
4. Joshua Holland writes that “[IFHS] was conducted by officials of the Iraqi Ministry of Health [...] The Moth was a Sadrist ministry, and that likely caused trepidation among many respondents.”
This is another widely-circulated attempt to “criticise” IFHS – and probably the weakest and least worthy I’ve seen, based as it is on misinformed assumptions and speculations. IFHS was peer-reviewed science, and well-received by leading epidemiologists. It was also, like the Lancet surveys, a very brave effort by the researchers in the field who risked their lives. One IFHS team member was shot and killed on his way to work. http://content.nejm.org/cgi/content/full/NEJMp0709003?query=TOC
5. Michael Schwartz writes that “Shone understates the results of the IFHS study”. In fact I don’t “understate” – I accurately state that IFHS “estimated 151,000 violent Iraqi deaths”. This “understates” no more than if I’d written, accurately, that Lancet 2006 estimated 601,000 violent deaths.
(Of course, I’m aware than both IFHS and Lancet 2006 covered only as far as Summer 2006, but I think it’s better to accurately represent survey estimates than to extrapolate to the present day based on one’s own fallible set of assumptions. I also think it’s better to accurately represent survey estimates than to attribute “total excess mortality” figures based on one’s own assumptions, or based on widely circulated, but false, attributions).
6. Joshua Holland writes that “total excess mortality attributed to the war was 400,000 in the IFHS study”. This is incorrect. The source of the “400,000” figure isn’t the IFHS study or the IFHS authors. IFHS doesn’t provide a “total excess mortality” estimate. It provides only crude death rates, and simple extrapolations from these to total excess deaths should not be imputed to it (for reasons made clear by the IFHS authors – eg recall issues could lead to a spurious “excess” figure without “further analysis”). [*See July 2010 footnote on the "400,000" figure]
(Ref: NEJM Volume 358:484-493 Number 5. The following is as far as the IFHS authors went in providing a “total excess” figure, ie not very far: “Overall mortality from nonviolent causes was about 60% higher in the post-invasion period than in the pre-invasion period. Although recall bias may contribute to the increase, since deaths before 2003 were less likely to be reported than more recent deaths, this finding warrants further analysis”. http://content.nejm.org/cgi/content/full/NEJMsa0707782)
7. Joshua Holland writes that the “400,000” excess mortality figure [falsely] attributed to IFHS is “a number that’s certainly in the neighborhood of the Lancet findings”.
It’s incomprehensible to me that Joshua thinks “400,000” is “certainly in the neighborhood of” the Lancet 2006 estimate of 655,000. There’s a difference here of over a quarter of a million deaths! Furthermore, the whole exercise of comparing “400,000” to 655,000 conflates violent and non-violent deaths, since only approximately 54,000 of the Lancet 2006 estimated deaths were non-violent (compared to 249,000 of the “400,000” figure falsely attributed to IFHS).
Even if one accepts the 400,000 “total excess” figure and overlooks the conflation of violent and non-violent deaths in the IFHS-Lancet comparison, it doesn’t remotely follow that IFHS corroborates the “one million deaths” figure.
8. Michael Schwartz writes that “While [ORB's] methodology is shakier than that of the Lancet studies, its results are consistent with their estimates.”
This alleged “consistency” between ORB and Lancet 2006 disappears upon close scrutiny. There are serious discrepancies in geographic distribution of violent deaths. ORB implies 700,000 violent deaths in Baghdad alone, compared to an estimate of about 150,000 implied by Lancet 2006 (requiring half a million violent deaths in Baghdad alone between July 2006 and August 2007 for the two studies to be “consistent”).
Moreover, there’s not even consistency between Lancet 2004 and Lancet 2006 on violent deaths. Lancet 2006 estimates twice as many violent deaths for the same period as Lancet 2004. (The Lancet 2004 estimate for violent excess deaths was 57,600; the Lancet 2006 data provides an estimate, for the period covered by Lancet 2004, of 112,500 -126,000 violent excess deaths, approximately twice that of Lancet 2004). Source for Lancet 2004 violent deaths estimate: Richard Garfield, Lancet 2004 co-author. http://www.epic-usa.org/An_Interview_with_EPIC_A.html
9. Michael Schwartz writes that he doesn’t accept, as a “lower bound”, the CRED figure of 125,000 deaths. But he provides no reason for dismissing the world-renowned and highly experienced epidemiologists at CRED. (Of course, I’m aware that the CRED figure, like Lancet 2006 and IFHS, covers only up to Summer 2006, and is by definition an “underestimate” relative to the present day – like all out-of-date studies).
Of all the published scientific studies, which do Michael Schwartz and Joshua Holland accept as providing a credible “lower bound”? And if Michael and Joshua are regarded as authorities by Project Censored, could this “lower bound” figure not be reflected in the Project Censored headline (eg “Estimates for Iraqi dead range from [approved lower bound] to over a million”)?
10. Joshua Holland writes: “I disagree, however, with the idea it [ORB's estimate] should be rejected because ORB’s ‘core competency’ is opinion polling.”
I agree. ORB’s findings shouldn’t be “rejected”, and nowhere do I suggest they should be. I merely point out possible reasons why ORB’s Iraq survey hasn’t been widely recognised or accepted in the scientific literature (with the exception of non-refereed writings by the Lancet authors, various blogs, etc). [July 2010 update - given the recent published criticisms of the ORB poll, I think its estimate should not be regarded as reliable]
I’d also add that while the peer-reviewed IFHS can be validly criticised, it seems to have been well-received by the majority of experts in the field – in contrast to the reception it received from, for example, John Tirman (who commissioned Lancet 2006). Tirman, who is no epidemiologist, was quick to dismiss the IFHS estimate of violent deaths as “not credible” (echoes of George Bush dismissing the Lancet study). http://tinyurl.com/3o8lsc
11. Michael Schwartz writes of “the second Lancet study, which remains the standard that I would accept for methodological rigor and therefore for accurate estimates”.
This is a curious statement, as the Lancet authors themselves are emphatic that they haven’t provided “accurate” estimates, but merely ballpark figures in need of corroboration. Michael also mentions the confidence interval – a wide range between lower and upper bounds on the estimates, and adds that this is “the correct way to report it in scholarly contexts”. Actually it’s the only correct way to report it in all contexts (including Project Censored headlines). As the revered demographer, Beth Osborne Daponte, points out, it’s a misinterpretation of the data in such studies to pick the mid-point of the range and present that as “the” definitive estimate. (Ref: the peer-reviewed ‘Wartime estimates of Iraqi civilian casualties’, by Beth Osborne Daponte, p7) http://www.icrc.org/Web/eng/siteeng0.nsf/htmlall/review-868-p943/$File/irrc-868_Daponte.pdf
12. Joshua Holland writes that the “IFHS study also revealed a pretty consistent number of deaths from year-to-year. That’s simply not plausible”.
This appears to be one of the more valid criticisms of IFHS. However, another possibility should be considered: that this only appears to be the case in the IFHS data due to the range of sampling error being substantial for the annual figures. The difference in the IFHS trendline and that presented by IBC is in fact not statistically significant.
Compare also the fact that Lancet 2006 data doesn’t show the spiking of violent deaths that you’d expect in the shock-and-awe phase (which is seen clearly in IBC’s detailed data). The “explanation” given by Les Roberts (Lancet co-author) for this should certainly raise some eyebrows: “Our data suggests that the shock-and-awe campaign was very careful, that a lot of the targets were genuine military targets”. Source for Les Roberts quote: http://tinyurl.com/4yo5uw
13. Joshua Holland writes that the “Lancet study measured excess mortality over a pre-war baseline from all causes — violent, nonviolent, etc. That means we have to compare apples and apples. Your correspondent cites the 151,000 figure from IFHS, but that was only violent deaths”.
Lancet 2006 in fact provided an estimate for violent deaths: 601,000 (the estimate of 655,000 was for deaths by all causes), so we can compare apples with apples (since IFHS also covered the same period as Lancet 2006). The direct comparison is as follows:
- Lancet: 601,000 violent deaths
– IFHS: 151,000 violent deaths
This is the most direct comparison available, and it shows a massive difference (450,000 violent deaths) between the two studies. Other comparisons are less direct and more questionable, as they introduce more assumptions, further extrapolations, etc.
*[July 2010 update. My comments on the false attribution of the 400,000 figure to IFHS remain accurate, but one of the study's authors has reportedly mentioned a similar figure, "397,000", several months after the study came out. To date, no details have been published - but it's possible that the IFHS authors will at some point publish a figure. Of course, this doesn't change the fact that when IFHS was published in January 2008, its authors specifically warned against attributing a figure to excess deaths, due to recall issues]
Unrelated update – The Comment Factory has published another of my articles: Dubious polls: How accurate are Iraq’s death counts?