ORB Iraq poll criticised May 3, 2010Posted by dissident93 in Iraq mortality.
Opinion Research Business (ORB) is a “corporate and issues-led market research” firm which received much publicity in 2007 when it estimated that over a million Iraqis had been murdered as a result of the Iraq war.
A poll on the complex issue of war-related deaths was untypical of ORB’s work (they’re opinion-pollsters not, for example, epidemiologists). More typical for them is the following type of “finding”:
Latest poll by ORB on behalf of BBC Newsnight reveals more people believe David Cameron and the Conservative Party will “make the right cuts in public spending” than Gordon Brown and Labour. (ORB press release)
ORB’s Iraq poll wasn’t peer-reviewed science (it was published nowhere but on the ORB website), and the person conducting it, Munqith Daghir, had little formal training or field experience (according to ORB’s publicity literature). Nevertheless, their figure of over 1,000,000 deaths was widely quoted as a serious estimate.
A new peer-reviewed paper (published by Survey Research Methods  Vol.4, No.1) details systematic errors (eg non-coverage and measurement errors) in the ORB Iraq poll. It shows that in four governorates in central Iraq (which account for more than 80% of ORB’s estimated one million deaths) a higher percentage of respondents report deaths of household members than, in an earlier ORB poll, reported deaths of extended family members. This cannot be seen as credible, since extended families are much larger than households.
There’s a response to this paper from ORB’s Johnny Heald (followed by a response [p2] to Heald by the paper’s authors). Heald doesn’t address the paper’s substantial criticisms. Instead, he adopts a rather defensive position with regard to ORB’s aims (Heald’s own emphasis):
The survey was only an estimate and the fundamental point of this and every other investigation into this subject remains the same i.e. there has been a very significant human cost associated with the conflict. […] Our findings were an estimate based on a survey – or opinion poll if it makes it clearer. We have repeatedly stressed that our work does only offer an estimate; again the key point is not whether we (or others) are 100% accurate…
Of course, nobody claims that it was anything but an “estimate” (or that “100% accuracy” was required). And few people disputed the “very significant human cost associated with the conflict” (in fact most of my friends and colleagues were calling it a “bloodbath” way back in 2003, based on Iraq Body Count’s figures). The point – which Heald doesn’t address – was whether ORB’s specific estimate of 1,033,000 dead can be considered remotely reliable, given the credibility problems with their data.
Heald then attacks the paper’s authors, asserting that they “had little intention of taking an objective view but are merely pursuing an agenda”. He then claims a “conflict of interest”, because one of the paper’s authors (Professor Michael Spagat) is, he says, “very closely linked to Iraq Body Count (IBC)”. Heald apparently overlooks the fact that the paper’s other author, Josh Dougherty, is a member of IBC, and that this is clearly indicated at the top of the paper – presumably the paper’s referees didn’t regard this as a “conflict of interest”!*
Perhaps if Heald spent more time studying how good scientific method works in practice, rather than reading conspiriological smear campaigns issued by “media criticism” websites (I speculate here, of course), he might have a better understanding of what “conflict of interest” means in the context of published research. I direct him to a paper criticising IBC, co-authored by Les Roberts (who was, you know, “closely linked” to the Lancet Iraq studies). How’s that for a “conflict of interest”?
It’s a pity that Heald resorts to ad hominem rather than addressing the points of substance raised by the paper. Quantifying war dead is a serious business. If you publish an estimate of 1,000,000 war-related deaths, based on a claim of a “nationally representative sample” (a difficult challenge in Iraq) then you should provide sufficient detail about the sampling methodology for your claim to be assessed. As the above paper states, “ORB’s parsimony with information about its methodology is an indicator of low survey quality and weakens confidence in its estimate.”
Low survey quality might not matter so much if you’re making trivial claims (for frivolous news reports) – eg “more people believe David Cameron than believe Gordon Brown”. But it surely matters if you’re publishing a figure of a million deaths resulting from a war, and this figure contradicts, by a large amount, the estimates of several other studies.
* Heald was the Conservative Party’s private pollster in the 2005 General Election (according to his ORB profile). If I had a “link” like that, I wouldn’t go around making vague, silly accusations of “conflict of interest”.
Related news: Prof Spagat’s long paper on ethical and data-integrity problems in the Lancet Iraq study (2006) has been published in Defence and Peace Economics. (Another of his papers, with Neil Johnson et al, made the cover of Nature journal recently. Perhaps Nature is part of the conspiracy? After all, it too was critical of the Lancet study). The editor of this journal notes that “The authors of the Lancet II Study were given the opportunity to reply to this article. No reply has been forthcoming”. That’s a shame, given the seriousness of Spagat’s criticisms.
• The 2006 Lancet Iraq study has been awarded the “STONEWALLING & COVERUP” Award in the 2010 Top Ten “Dubious Polling” Awards from StinkyJournalism.org (a Media Ethics Project of the Art Science Research Laboratory). I checked the background of the two people who decided the awards (George F. Bishop and David W. Moore), to make sure they’re not “closely linked” to anything suspiciously like a “conflict of interest”. They passed the test – they’re not rightwing warmongers or anything.