More damed lies?

Yesterday, the UN Assistance Mission for Iraq (UNAMI) released its human rights report for November-December 2006. It claimed,

For 2006, the total number of civilians violently killed is 34,452: 16,867 from the Medical Legal Institute in Baghdad (unidentified bodies) and 17,585 from hospitals (operation centres) throughout Iraq. The yearly average is 94 civilians killed every day.

Estimates of the numbers of victims of the US led invasion and occupation of Iraq have always aroused controversy, and this one promises to be no different. In fact, Dr. Hakem al-Zamili, Iraq's deputy health minister,

told The Associated Press the United Nations may be using unreliable sources for its casualty count. "They might be taking the figures from people who are opposed to the government or to the Americans," he said. "They are not accurate." He said he would provide Iraqi government figures later this week.

In early January, a compilation of Iraqi government figures put last year's civilian deaths at just 12,357.

Meanwhile, Iraq Body Count claims a total of civilian deaths for 2006 of between 19,535 and 21,036, half again as high as the Iraqi ‘government’ figures, but much lower than the UN claim. The Lancet study published on 11 October only estimated deaths through the study enumeration period between May and July last year, so there is no estimate for either the November-December 2006 period, or for 2006 as a whole.

To complicate matters, the UN does not appear to have published a total of all deaths since the invasion. For the purpose of comparison, assuming Iraqis have died at a constant rate over the period since March 2003, that would come to a total of about 132,000. As the rate has been accelerating, this total is necessarily high.

It is worth looking at the methodologies used to calculate each figure to achieve a better idea of which might provide a closer representation of the actual scale of destruction.

The UNAMI report does not discuss methodology. The only relevant information is the footnote on page 4,

Figures of civilians violently killed and wounded are based on the number of casualties compiled by the Ministry of Health from hospitals throughout the country and the Medico-Legal Institute in Baghdad. It should be noted that for the month of December, figures from some Governorates were not yet included in the total provided.

So this report is presenting not a statistical estimate, but an actual count based on reports from two Iraqi government sources. It is widely understood that the Iraqi government is constrained by the fact that the country it is supposed to govern is actually occupied by scores of thousands of foreign troops armed to the teeth. If that didn’t complicate the collection of administrative data of this kind enough, the insurgency that has arisen in response to the occupation largely regards the government and those cooperating with it as tools of the occupation, making collection of government data even more difficult and dangerous. It is curious that the Ministry of Health itself, one of UNAMI’s two sources, disputes UNAMI’s figures.

Last month, Ashraf Qazi, the senior United Nations envoy to Iraq, ‘cited statistics illustrating the stark problems facing Iraq, where more than 5,000 people die violent deaths each month’. Magazzeni reports, ‘According to information made available to UNAMI, 6,376 civilians were violently killed in November and December 2006’ [my emphasis]. So Qazi’s claim is nearly 40% higher than the more recent count. A possible source of the discrepancy is that Qazi is counting all persons, while Magazzeni is only counting specifically civilians specifically killed in sectarian violence. My inclination is to doubt that this distinction truly accounts for the discrepancy, because it is very common for reports of this kind to use undefined terms loosely. I suspect that on the one hand, Qazi was pulling a number out of a hat rather than relying on specific sources and calculations, and that Magazzeni’s sources can not reliably distinguish either civilians from combatants or victims of sectarian violence from, say, criminal violence.

Iraq Body Count, in contrast, is very explicit about exactly what they count. According to their methodology page, IBC provides a count of

media-reported civilian deaths in Iraq that have resulted from the 2003 military intervention by the USA and its allies. The count includes civilian deaths caused by coalition military action and by military or paramilitary responses to the coalition presence (e.g. insurgent and terrorist attacks).

It also includes excess civilian deaths caused by criminal action resulting from the breakdown in law and order which followed the coalition invasion.

It is simply a count of deaths reported in at least two independent media sources:

For a source to be considered acceptable to this project it must comply with the following standards: (1) site updated at least daily; (2) all stories separately archived on the site, with a unique url; (3) source widely cited or referenced by other sources; (4) English Language site; (5) fully public (preferably free) web-access.

The project relies on the professional rigour of the approved reporting agencies. It is assumed that any agency that has attained a respected international status operates its own rigorous checks before publishing items (including, where possible, eye-witness and confidential sources). By requiring that two independent agencies publish a report before we are willing to add it to the count, we are premising our own count on the self-correcting nature of the increasingly inter-connected international media network. http://www.iraqbodycount.org/background.php

The site lists 38 sources identified as ‘some core sources’. It is not at all clear whether this list is comprehensive or indicative, and if not exhaustive, what proportion of all core sources it comprises. Nor do we know whether non core sources are being used, which ones, when, or on what basis. Anyway, among the ‘core sources’ they mention are such beacons of truth as Fox news, the London Telegraph, the Toronto Star (but not the slightly more reputable Globe and mail), and of course the NYT.

When IBC write of ‘two independent agencies’, they can only mean that the reports come from completely independent sources. So if the NYT reports what AFP said, that should not count as a second report. Ultimately, it seems to mean that at least two different reporters have to have interviewed witnesses or seen documentation of a death before it adds to the IBC count. We know that the vast majority of reporters in Iraq are either holed up in the Emerald City far from the action or embedded with the occupation forces. Some reports may exist that are never published for editorial reasons. Under the circumstances, it is hardly surprising that only a fraction of the total number of deaths that actually occur are reported, and an even smaller fraction independently reported by two separate agencies.

IBC rely on the ‘professional rigour’ of the mainstream media not only for numbers but, like the UN sources, to determine whether a corpse was in life a civilian or a combatant. It is notoriously difficult to make this distinction in the kind of counterinsurgency that’s going on in Iraq and I think it must be particularly difficult in a society where nearly everyone is armed.

The Lancet study has its drawbacks. As a survey rather than a count like the administrative data the UN rely on or the media reports IBC uses, the number arrived at is inherently fuzzy. Because of the necessarily small sample, in this case, the authors are 95% confident that the actual number of violent deaths lies somewhere between 426 369 and 793 663. That’s not very accurate, and the authors are honest about it. It’s important to remember that a statistical 'estimate' is not just a guess - it is calculated by multiplying the number of actual observations by the weights, i.e. the proportion of the population represented in the sample.

Furthermore, the cluster sampling method adopted is unlikely to provide the level of accuracy we’d expect from an ordinary sample survey. For logistical reasons, however, this is the universally accepted method of sampling in studies of this kind in areas that present dangers to the collectors. The definition of a household the researchers used is not awfully satisfactory, but is very unlikely to have impacted on estimates. The total population of Iraq is not known with great accuracy and this does impact on the calculation of the weights by which the actual observations are multiplied, but this is the case wherever robust systems for recording births and deaths are absent, which means most of the world, and statistics from, for example, Pakistan, are not treated as controversial. Unlike the UN, Iraqi government, and IBC counts, the Lancet study does not pretend to distinguish combatants from non combatants.

There are two important things to note. First of all, the lowest probable number of estimated deaths is much higher than the deliberately high extrapolations I calculated from the UN data, which come to some 127,500 on the basis of the Magazzeni report and 200,000 on the basis of Qazi’s figure for the forty months to July, and nearly ten times the IBC figure of 43714/48556 for July 2006. (Note that the IBC minimum and maximum figures do not represent the extremes of a statistical confidence interval, but are separate counts arrived at where sources are inconsistent.)

The second thing to bear in mind is that the probability that the actual number dying from violent causes over the relevant period is exactly as likely to be 793,663 as it is to be 426,369. There is no reason to consider the lower estimate more plausible. That said, based in part on the reasoning presented by Eli Stephens, of the Left i on the news blog, if we are 95% certain the true figure is in the range 426 369 - 793 663, then we are more than 95% certain that it is at least 426 369.

In summary, then, we are more than 95% confident that, as of July, well over 400,000 Iraqis fell pray to the violence deliberately unleashed by the US led invasion. In contrast, we are 100% certain that the quisling Iraqi government’s own figures are likely to be undercounts, even in the implausible eventuality that they are not being cynically tampered with. These in turn form the basis for the UNAMI figures, which are therefore suspect. We are also 100% confident that the media can not possibly know of all Iraqi deaths and IBC’s extremely cautious methodology ensures that the number will in any case be minimized.

Even if the government and media sources could really reliably distinguish a civilian body from a combatant body and did so, it would not account for the discrepancy between their counts and the Lancet study estimates. It’s also worth pointing out that most of the ‘combatants’ killed in the counterinsurgency – those who really are combatants – are only in combat because of the invasion and occupation that they are quite legitimately resisting. So attempting to exclude them from the war toll is deeply cynical.

Speaking from Kuwait, US Secretary of State Condoleezza Rice, in an uncharacteristically honest moment, said,

"Violent people will always be able to kill innocent people," she said.…"But whatever the number of civilians who have died in Iraq - and there obviously are competing numbers - but whatever the number is, it's too many," she said.

If only she and her violent mates would stop killing innocent people.