Tim Thurley: Our gun-control proposals have a very basic failure — garbage data

The Mass Casualty Commission relied on skewed (at best) data and its gun conclusions cannot be taken seriously.

On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

— Charles Babbage

By: Tim Thurley

The Mass Casualty Commission’s firearm recommendations were, rightly, overlooked in the initial phase after the report’s release. They have become relevant these past weeks as gun control groups, the NDP, the Bloc, and the Liberals used them to advocate for sweeping changes to Bill C-21, the government’s controversial gun-control proposals. The Liberals have thus far declined to adopt the MCC’s recommendations, at least in whole, and that’s encouraging. Our lawmakers should be careful. The Mass Casualty Commission’s concluding recommendations on guns and homicide share a problem common to any data analysis. If you use the wrong data, you get a bad output. 

Or, to be blunt: garbage in, garbage out.

R. Blake Brown, a professor who contributed a commissioned report to the MCC, suggested that the MCC got all the best research together and simply found the arguments made by gun control groups to be more convincing.

He’s wrong. While the MCC could have been a completely neutral panel objectively weighing the evidence before it, the nakedly selective choice of data inputs and slanted interpretation meant that no unbiased outcome was possible. Indeed, the MCC inputs seem heavily weighed to advance a pro-control agenda, and do so in such an obvious way that the resulting flaws should be immediately clear to those with even a passing knowledge of the study of firearms and firearm homicide.

The Commission relied on witnesses, third-party testimony and a series of 23 reports and various jurisdictional scans of varying relevance to gun control. But the core of its justification for gun-control recommendations on violence prevention grounds seems informed, quite narrowly, from a report it commissioned and paid for: the Negin Report. If that report were representative of the full academic gun debate, then all would probably be fine.

It isn’t.

The Negin Report was authored by Joel Negin, Philip Alpers and Rebecca Peters, at least two of whom have been involved with Australian gun-control efforts. Peters was a founding member of the National Coalition for Gun Control in Australia (later known as Gun Control Australia); Alpers is a frequent advocate in the press for stronger gun-control measures. An excellent selection of researchers and advocates to make a pro-control argument, to be sure, but perhaps not one best aligned with the “non-partisan” mission of the Commission.

Of course, we all have personal biases, and they’re not a sufficient academic argument. Peters and Alpers are entirely entitled to their views — I do not quibble with them there. So let’s get to the problems with the report itself.

This is going to get a little dense. You’ve been warned.

At the core of the Negin Report’s assumptions is the incorrect but frequently-made claim that sweeping Australian gun control changes in the form of the National Firearms Agreement (NFA), which included a general ban on semiautomatic rifles, was unquestionably responsible for a dramatic reduction in gun homicide and mass shootings. To arrive at this conclusion the Negin Report ignores or misrepresents the research, makes misleading claims regarding multiple studies — including, rather impressively, their own — and fails to recognize other important potential causal factors that may have contributed to a decline in homicide rates.

The Negin Report relies extensively on papers such as those by Chapman, Alpers, Agho, and Jones (2006) and Chapman et al (2016). These papers are generally fine, albeit with limitations. A major problem is that the Negin Report, alongside many gun-control advocates, uses them to make claims the papers themselves just don’t back up. While citing them as showing a decline in firearm mortality after the NFA was passed, the report authors fail to mention the most important bit of the studies: they failed to reach statistical significance. In the 2016 study’s own words, “it is not possible to determine whether the change in firearm deaths can be attributed to the gun law reforms.”

This means that within the bounds of those studies the Australian laws cannot be associated with, or shown to be the cause of, a decline in firearms homicide rates. Even some of their own studies, those studies making the backbone of their case, provide evidence that the Australian laws likely didn’t reduce firearm homicide.

Negin et al mention a battery of thorough statistical analysis studies — many of which found no effect from the Australian laws — exactly once, in a text box on page 20, and then only to dismiss them. Their limited grounds for dismissing these studies are very weak. The report claims that these analyses cannot be valid because it is impossible to run a control in a real-world scenario, even though that is the exact situation those research designs are designed to account for. It then quotes Hemenway, an American researcher, to say that any policymaker would consider the observed reduction to be a success even if the reduction couldn’t be shown as being attributable to the law. 

If this is true, and it probably is, it says rather more about the poor quality of modern policymaking than the evidence itself. I would suggest that if a problem is so infinitesimally small that we cannot statistically isolate the impact of even the harshest measures to control it, perhaps imitating those ineffective policies by spending billions upon billions of dollars and threatening peaceful citizens with prison in an attempt to “solve” it might be just a slight overreaction.

The Negin Report fails to substantively engage with the most relevant critiques of these massive gun control measures. There is no coherent discussion, and in fact but a single mention, of method substitution, a critical concept; that homicide rates were already declining at a steady pace prior to the interventions; that there was no real deviation from the pre-existing trend; that the decline in death rates was actually steeper prior to the NFA; or that non-firearm homicides also declined at a fast rate or indeed a faster rate, which suggests other reasons for the decline than gun bans. 

Firearm homicides and firearm deaths (which include accidents and suicides) are regularly discussed, but are sometimes conflated despite these being different problems with different pathologies and different solutions. Overall homicides are, of course, the actual relevant figure, unless we merely care about how people have been murdered and not that they are dead.

Yes, the Mass Casualty Commission Report was about mass shootings and not necessarily about homicide as a whole. The problem is that the Negin Report does not treat its analysis of these mass casualty events any better than it treats its analysis of homicide. By claiming that there were 13 mass shootings (when only two were likely relevant to the NFA changes) in the years prior to the NFA and none thereafter (depending on definitions, there have actually been at least three) and lauding those unexamined numbers as a clear success, the Negin Report fails on essential grounds.

Let me expand. Mass shootings and homicide events are so incredibly rare outside of the U.S. and susceptible to contagion effects that it is difficult to establish causality or statistical significance in any post-legislation rise or decline. The rare-events model used to get around this inconvenient problem is contained in a 2018 paper authored in part by (who else?) Chapman and Alpers, which claims that without the NFA Australia should have expected 16 mass shootings from 1996 to 2018. 

So why is it irresponsible to rely on that model to claim the NFA had an impact on mass murder?

Leaving the methodological arguments aside — and I have a few — not only were mass shootings relatively rare in the period prior to the 1980s and 1990s, but many of the 13 cited would not have been plausibly prevented by a semi-automatic rifle ban. At least six of the 13 were spree shootings carried out slowly, like Portapique. This means they could have been done with essentially any firearm or indeed any weapon, such as the knives that killed 11 innocents at James Smith Cree Nation. Of the remaining seven Australian mass shootings, five were committed with firearms that the NFA did not ban, which remained legal in the intervening period and are legal today. 

In sum: if the semi-automatic rifle ban had been implemented earlier it likely would not have even impacted 10 or 11 of the 13 mass homicides.

It is prudent to note that the NFA was not just a semi-automatic rifle ban, but a broad package of gun controls. If the NFA can be attributed to be the causal event via gun bans — which, as we have established above, it likely can’t — surely it is more reasonable to believe that other factors which have been considered effective by other research, such as waiting periods, strict licensing requirements, and strict storage requirements — all of which are in place in Canada — would have been the main causal factors. Attributing them to the semi-automatic rifle ban, which we know from most relevant research has no real impact on homicide or mass homicide writ large, is a spurious argument.

Further worth noting is that the mass shooting model only applies to mass shootings, and not to overall mass homicide reduction, such as the vehicular, stabbing, and arson mass homicides which took place in Australia in the 2000s and 2010s. This matters. Should we care more about how people are murdered than whether they are dead? Of course not. It is overall violent death rates and mass homicide rates which matter, not firearm violent death rates, and any intervention must be judged to that standard.

I could go on. The Negin Report is 45 pages long, and I have limited space to critique every flaw. The serious flaws identified above are critical, and sufficient to show the inherent bias and unreliability in the data used by the Commission. That is not to say that the Report does not articulate relevant research; rather, by calling the Australian experience a “notable public health success” it misrepresents what the research demonstrates: that the balance of evidence does not show that the NFA reduced firearm homicide, overall homicide, or mass homicide rates. If the Commission was presented with such a one-sided analysis, how could it possibly make a balanced recommendation?

The Negin Report does not address — and indeed is not meant to address — Canadian research. That’s fine. My criticisms of the report aside, the Commission was absolutely entitled to examine non-Canadian research to see if it could find solutions elsewhere. What was inexcusable was for the Commission to flatly ignore the substantive body of critical research, including Canadian research, which does exist. That was an egregious failure, and one that should relegate its recommendations to the dustbin.

Dr. Caillin Langmann is a well-known name in Canadian firearms research, and by far the most prolific author using rigorous statistical methods to examine the effects of gun control on Canadian firearm mortality. No serious analysis of Canadian firearm mortality is possible without his work, and yet his work does not appear on its own and is not cited in the Negin Report. Indeed, his and other critical research does not seem to have informed the final Commission report or recommendations at all.

I asked Dr. Langmann about his exclusion. He told me he offered to appear to present his research but the Commission declined. 

It may not be a coincidence that the exclusion of Langmann and other researchers without explicit gun-control agendas was due to the fact — the fact — that the Canadian and comparable research substantively contradicts the Negin Report and the MCC recommendations on firearms. An examination of already-implemented Canadian gun laws including various factors such as prohibition of “paramilitary style” rifles and magazine capacity restrictions all found no impact on mass shootings or mass homicide overall in Canada or on associated fatalities. Instead, mass homicide by both firearm and non-firearm causes gradually declined on its own. The lack of association between gun control and decreased mortality is replicated multiple times in Canadian research.

Guess what? It is also replicated in a detailed statistical analysis of Australian data not mentioned by the Negin Report.

The core research inputs to the Mass Casualty Commission were commissioned from parties with well-established and acknowledged positions on firearms. Written by literal gun control advocates without substantial input from other sources, the contrary research is either ignored or not treated with due academic respect. This damages the credibility of the Commission findings, giving the perception that they were gathering conclusions in search of evidence. 

Again, it must be made clear that this wouldn’t have been a problem if the MCC had treated the Negin Report as just one part of the firearm policy research puzzle. It was their failure to do so and the consequent lack of neutrality, lack of engagement with solid research, and complete disregard for engagement with different academic perspectives despite obvious relevant expertise, that taints the Mass Casualty Commission firearm recommendations and severely limits any useful policy conclusions we can gather from their report.

Given the poor quality of the inputs, it is a small wonder that the new gun control recommendations to be discussed in Part Two of this piece were so utterly detached from reality. Garbage in…

Tim Thurley specializes in firearm policy, having earned a Master of Science from Leiden University with his analysis of the long-gun registry’s lack of effect on Canadian homicide rates. He lives in the Northwest Territories, where he files regular Access to Information requests on firearm issues and anything else of interest.

---

The Line is Canada’s last, best hope for irreverent commentary. We reject bullshit. We love lively writing. Please consider supporting us by subscribing. Follow us on Twitter @the_lineca. Fight with us on Facebook. Pitch us something: lineeditor@protonmail.com