Toll dwarfed?
A few days ago, lenin had a piece quoting extensively from Anthony Arnove’s article on Tomdispatch drawing a comparison between the treatment of statistics of the tragedy in Darfur and in
Since 2003, according to UN estimates, some 200,000 have been killed in the Darfur region of
How would you know this? Well, if you lived in
The point of this post is not, however to discuss the reasons for the discrepancy between all the noise about Darfur and the silence about
One of the ironies is that the emphasis on the
UNHCR estimates there are some 1.9 million Iraqis displaced internally, and up to 2 million in neighbouring states, particularly
In other words, the
What really prompted this post is that while Arnove and others continue to cite the Johns Hopkins study estimate of 655,000 excess deaths published in the Lancet last October, Information Clearinghouse posted an article entitled ‘Deaths In Iraq Have Reached 1 Million’, dated 22 March. Attributed to an ‘Alan Jones’ but not linked to any source, it starts out by claiming,
THE number of deaths in
On the fourth anniversary of the invasion by Allied troops, an Australian scientist insisted the true death toll dwarfed previous estimates.
Dr Gideon Polya said: "Using the most comprehensive and authoritative literature and UN demographic data yields an estimate of one million post-invasion excess deaths in
The article does in fact link to Dr Polya’s Global Avoidable Mortality blog, where the most recent post, dated 16 May 2006, relates to Isr
In reality, the ICH article appears to relate not to anything published on 21 March, but to Dr Polya's 1 March editorial on his Media With Conscience site. He actually presents the same analysis in a 7 February Countercurrents article.
In the 7 February article, Polya writes, ‘the post-invasion excess deaths (avoidable deaths, deaths that did not have to happen) total 1.0 million (ONE MILLION)’. This is his reasoning,
Consider the following estimate from the Johns Hopkins medical scientists of "annual death rate per 1,000 of population" of 13.3 (post-invasion
The "post-invasion excess death rate/1000 of population" was 13.3 - 5.5 = 7.8 (Comparison A) or 13.3 - 4.0 = 9.3 (Comparison B). Assuming an average population of 27 million, the "post-invasion excess deaths" total (over 4 years i.e. as of February 2007) (A) 7.8 x 2,700 x 4 = 842,000 and (B) 9.3 x 2,700 x 4 = 1,004,400 i.e. ONE MILLION.
In other words, the estimate of one million is arrived at by multiplying the difference between the Johns Hopkins study’s estimate of deaths per 1000 in
As far as I’m concerned, it’s not really outrageous to compare the post invasion Iraqi crude mortality rate with the rate for a neighbouring country. The rationale is that preinvasion mortality in
I’ve made attempts to update the Lancet estimate before, but the Lancet data are now eight months old and we know the slaughter has proceeded, by all accounts at an accelerating pace, we need some new estimates. The report of Polya’s estimate has prompted me to take it a little further. This is not to cast aspersions on Polya’s method, but I have adopted a different approach.
The Lancet estimates published last October cover the period March 2003 through June 2006. The estimated average monthly crude death rate for the fourteen months through April 2004 was 7,017. For the thirteen months from May 2004 to May 2005, the average was 15,115 per month, and for the last thirteen months June 2005 to June 2006, 27,710.
The method I am using is to assume that the Lancet estimates are correct as a starting point and increment them for the eight months since July 2006 based on monthly averages. I have calculated projections using three assumptions. As I am not actually weighting new data, but just projecting from the July 2006 data on the basis of some assumptions about the probable average monthly rate of increase, I will round the figures to the nearest thousand.
First, if we assume that the monthly rate of deaths over the last eight months was equivalent to the average of the forty months from March 2003 to July 2006, 16,373 per month, the current toll would stand at about 786,000. I regard this as highly improbable, as it would mean that the average monthly crude death rate had decreased by 41% from the 2005-2006 average.
Second, if we assume that the monthly rate has remained at the average for the 2005-2006 period, the total would now come to 877,000.
Third, if we assume that the current average monthly crude death rate has increased proportionally by as much over the 2005-2006 average as that average increased over the previous thirteen month period, by about 83%, the monthly average would now be 50,801 and the total to date, 1,061,000.
Bear in mind that these are based on statistical estimates each of which represents the approximate midpoint of a range of values. Principally because the size of the sample in the Johns Hopkins study, the confidence interval is wide. The claim the authors, Gilbert Burnham, Riyadh Lafta, Shannon Doocy, and Les Roberts, made was that they were 95% confident that the true number of excess deaths in July was between 393,000 and 943,000. Using the same confidence interval proportionally, in the first and least likely scenario, where the monthly rate had decreased, we would be talking about a range of 472,000 to 1,131,000. In the second scenario, where the rate had remained constant from last year, it would be between 526,000 and 1,262,000. Finally, if the average monthly crude death rate has increased as much as it did between 2004-05 and 2005-06, the range is 637,000 to 1,528,000.
The following table may clarify the results.
| | | Estimate | 95% Confidence interval | |
| | | | Minimum | Maximum |
| Lancet estimate to 2006 07 | | 654,965 | 392,979 | 942,636 |
| Polya’s estimate | | 842,000 | n/a | n/a |
| | | 1,004,400 | n/a | n/a |
| My projection assuming increase at: | Per month | | | |
| A. 2003-2006 average rate | 16,373 | 786,000 | 472,000 | 1,131,000 |
| B. 2005-2006 average rate | 27,710 | 877,000 | 526,000* | 1,262,000 |
| C. 83% above B. | 50,801 | 1,061,000** | 637,000 | 1,528,000 |
* The true death toll is almost certainly more than 526,000.
** The probable death toll is 1,061,000.
From all reports, the rate of increase has been increasing itself, so even these highest projections are likely to be on the low side. Conservatively speaking, I think you could claim with nearly 100% confidence that the crude mortality rate over the last eight months has certainly not decreased from the average of the previous thirteen month period and that the US invasion and occupation has cost at least 525,000 Iraqi lives over the last four years. If, as seems probable, the CMR has in fact increased since last July, the figure is very likely to be over a million and could exceed a million and a half.
The central point is that, horrific though the situation is in Darfur, estimates based on similar methods demonstrate that it is in fact much much worse in Iraq, where the US, UK, and Australian governments bear direct responsibility for the catastrophe.
An ancillary point is how we use these statistics. At the top of the daily ICH email and the ICH site it asserts, ‘Number Of Iraqi Civilians Slaughtered In America's War On Iraq - At Least 655,000 + +’. I thought that the
What they actually found was that there were 654,965 excess deaths in
For one thing, if there is an ‘at least figure, it is not 654,965, but 392,979. For another, the Johns Hopkins study is absolutely explicit that, ‘Separation of combatant from non-combatant deaths during interviews was not attempted…’ So their research provides no basis for any claim specifically about ‘civilians’. Finally, these estimates are for deaths from all causes. The relevant estimate of specifically violent deaths to July 2006 is 601,027 (i.e. in the range 426,369–793,663). Presumably, that would be the number slaughtered.
It’s not as if the Lancet study’s findings were not sensational enough. There’s no need to distort them as ICH continues to do, much less to headline Polya’s 1 million figure, which explicitly includes sanctions effects with the invasion and occupation.
In my view, the safest, most responsible approach remains to be to cite ‘the July 2006 estimate of about 655,000’. Alternatively, as Eli Stephens of Left I on the news points out, the confidence that the true figure is above 393,979 is nearly 98%, so it would be even more accurate to assert that the invasion and occupation of Iraq killed ‘at least 394,000’ by July 2006. However, I am fairly comfortable that my projections provide a sound basis for asserting that the current death toll definitely exceeds half a million, is probably much higher, and possibly three times that number.
Thanks for this careful analysis. If you want to split even more hairs, you might have to consider the total population of Iraq vs. that of Darfur. I do, however, completely agree with lenin's, and Herman's, political point.
ReplyDeleteThanks, Christian. You're right, of course, that in some sense, some might consider the proportion killed as important as the absolute number. In strictly humanitarian terms, though, I think the absolute number is what's significant.
ReplyDeleteUnfortunately, things that we take for granted in countries that have a well established statistical infrastructure and routine, regular recording of vital statistics, like knowing the population with some accuracy, are luxuries elsewhere.
I haven't seen the kind of sound reporting of methodology underlying the Sudan mortality estimates as the Johns Hopkins group provided for Iraq. I understand that the surveys were conducted among displaced persons in camps. If so, there's no knowing how that would skew the results. Nor how not surveying the refugee population from Iraq would skew those results.
Bearing in mind that basically everything is in doubt, if the population of Darfur is, as Wikipedia asserts, 7.4 million, and the death toll has been 200,000, that would come to some 2.7% of the population. A horrifying proportion. If the population of Iraq is 27 million, notwithstanding the 2 million refugees, then if I am right in thinking my projection of 877,000 is the number killed so far, that would be some 3.2%. If the actual numbebr killed is the minimum I projected, 526,000, that would be 1.9%. One way or another, we're looking at two disasters of epic proportions and the point remains that one is a celebrity issue on everyone's lips. And the other, the one that the US government is directly responsible for, the one that is much worse in absolute terms and probably even proportionally, is swept under the carpet.
I don't know what happened to my table. I must go and fix that.
There's little I can say to fault Ernie's methodology, except to say that I'd expect the confidence interval to widen somewhat further than proportionally because of the effect of his extrapolation. I don't have statistical qualifications, so I won't go further than that. In any event, the death toll so derived is huge. Its credibility rests, though, on the Lancet study.
ReplyDeleteThe Lancet study seemed to me to have a pretty good methodology from the angle of statistical mathematics. My doubts are from a different direction - that of statistical collection and processing. Household surveys are notoriously difficult to get answers from, and reliable answers are even more difficult. National statistical organisations put massive efforts into questionnaire design and interviewer training in order to get half-way decent figures even in industrialised countries (Leading questions are the biggest source of non-sample error in many surveys - and it's often not deliberate). Getting frank and accurate answers to household survey questions in Iraq would be, to put it mildly, challenging. And, on top of this, interpreting them would require understanding Iraqi household structure sufficiently well to be able to translate raw answers per household into meaningful statistics for individuals at the aggregate level.
Ideally, I'd like to see the crew who did the original Lancet study do a follow-up, addressing whatever weaknesses in the original that the critics have identified and can be remedied in the circumstnces. If they don't have somebody on board with experience in household surveys, adding someone of that background would definitely assist.
Thanks for your comment ABIM.
ReplyDeleteAs you probably know, the interviewers in this survey were Iraqi doctors. The study isn’t explicit about the question wording or interviewer training. Ordinarily, I would say that statistics are meaningless without that kind of metadata. But then there would hardly be any statistics left. So in this case I assume that the questions, which were not particularly challenging to word, were ok and that the interviewers had enough training at least to know that they’re supposed ask the questions as worded and so forth. In reality, we know that even highly trained and experienced interviewers alter question wording arbitrarily if they find respondents don’t understand them or they’re too much of a mouthful.
There are many sources of non sampling error. As it’s very elusive and statisticians don’t like to talk about it, I’m not sure how you know that the principal source is leading questions.
Obviously, leading questions are a serious issue in opinion polls and the like, but I haven’t seen much of that in regular official surveys, and as I think you know, I read a lot of survey questionnaires. You do find other problems, like ambiguous syntax, use of unfamiliar terms or terms used in technical senses, infelicitous sequencing, and so forth. But not much in the way of leading questions as I understand the term. In any case, if, as you say, ‘Leading questions are the biggest source of non-sample error in many surveys - and it's often not deliberate’, why do you reckon ‘somebody on board with experience in household surveys’ – the very people who inadvertently include leading questions - would help?
I think in many cases, the main source of non sampling error in many countries is that the enumerators just sit under a tree and tick off answers rather than actually interviewing in households. I expect that the Iraqi doctors would have been more professional than that. Other sources are that respondents may misreport either from simple error, memory failure, distrust, boredom, or malice. When it comes to pinpointing the time of an event, there are known inclinations to favour round numbers of months or years and so forth. But although there might indeed have been factors conducive to deliberate or inadvertent misreporting in this survey, the report specifies that when asked, most deaths were documented – not only the fact, but the date and cause.
As I’ve written somewhere or other, my main concerns are the household definition and the cluster sampling methodology.
‘The third stage [of sampling] consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed. For this study, a household was defined as a unit that ate together, and had a separate entrance from the street or a separate apartment entrance.’
I’m pretty confident that the interviewers did not take the steps required to establish that all the household members enumerated actually ate together. The reason I’m so confident is that many national statistical agencies have used similar definitions and most have moved to a strictly usual residence approach to household definition precisely because they did not and could not establish eating arrangements.
Because households residing in adjacent dwellings are a priori more likely to be similar to each other in some significant respect, I think it is definitely preferable to sample dwellings at random, or technically, to use a skip between selected dwellings greater than zero. But cluster sampling is established as the preferred methodology not only in areas of conflict, but even under more propitious circumstances. For a number of years, UNICEF, for example, has had a program of Multiple Indicator Cluster Surveys (MICS) that they conduct in various countries (http://www.childinfo.org/mics/mics3/manual.php). There are usually cost and logistical factors that make it a more feasible option than a random sample at the final stage of sampling. But I haven’t read of anyone rejecting the MICS results on the grounds that it samples clusters. They try to compensate for the cluster selection at the final stage of sampling by random sampling at higher levels of selection. (Vacant dwellings and refusals were just skipped, another minor methodological concern.)
The main criticism of the Lancet study has come from an English economist named Michael Spagat who appears to have some quite unsavoury connections in Colombia. The nature of the objection is what they call ‘Main Street bias’. The claim is that the streets that intersect main streets are themselves main streets and likely to be where markets are located and consequently the most probable venues for car bombs and other violence. It’s true that the street layouts in Iraqi cities can be complex. And I don’t think we know which streets the interviewers selected.
There are two problems with the ‘main street bias’ hypothesis. For one thing, Spagat et al are assuming that the street where the enumeration started is a main street intersecting a main street. That may or may not be. It depends on how the enumerators interpreted ‘main street’, and I doubt if we know that. But, since they were specifically supposed to be sampling a ‘residential street’ intersecting a main street, I would expect that it would be neither a primary artery nor the kind of secondary main street that concerns Spagat. Furthermore, as it is supposed to be specifically a residential street, presumably it would be less likely to be a street with a bazaar, even street vendors. Also, dwellings on intersecting streets may have been included in the cluster, if the interviewers turned a corner rather than crossing the intersecting street, as they may have done. But more importantly, few of the deaths actually enumerated were at home, so it should not matter where the dwelling was located. People living on side streets are just as likely to get blown up or shot at a bazaar on a nearby main street as the people who actually live on that street.
The main point remains, however, that the methodology deployed in the Burnhan et al. study is universally acknowledged to be state of the art. When applied outside Iraq, the very people who purport to doubt Burnham welcome the estimates uncritically.
I don’t have a great deal of confidence in my projections. But I am confident in the original Lancet estimates and I’m quite sure that the numbers have increased a lot in the last nine months. While the firsts death in the invasion was already one too many, it does matter how many more than that the occupation has caused.
Ernie's right about the main source of non-sample error. I'd forgotten about the "sit under a tree & estimate it" approach, since it's not prevalent in Australia - at least since they stopped getting the coppers to collect the Agricultural Census, and that was maybe 40 years ago. I would, however, note that I was using the term "leading questions" rather loosely, so it included "ambiguous syntax, use of unfamiliar terms or terms used in technical senses, infelicitous sequencing, and so forth".
ReplyDeleteAnd, while statisticians don't like to talk too much in public about non-sample error, I've heard enough of them in private to know how badly poor questionnaire design (which is probably a more accurate term than "leading questions") will stuff up results. The worst effects normally come when the survey is designed by people who are experts in the subject matter under examination, but not in household surveys.
Finally, with the added information from Ernie above, I'm now pretty happy with the Lancet study. From previous discussion, I was aware of the problems with cluster sampling, but as far as I'm concerned that is something that is openly structured into the confidence interval.