The Florida State Senate this week heard arguments regarding a proposal to allow concealed weapons to be carried on the campuses of Florida State University. While the idea of guns on campus is unthinkable to most Europeans, several US states (Arkansas, Colorado, Idaho, Kansas, Mississippi, Oregon, Utah and Washington) technically allow it (though, in some cases, individual schools can impose their own bans). The issue is particularly contentious at FSU, which saw shooting incidents in both 2011 and 2014.

The debate is a familiar one. Supporters of the proposal, such as the president of Florida Students for Concealed Carry, Erek Culbreath, argue that while it’s impossible to keep guns away from the bad guys, “you can level the playing field by giving law abiding citizens a chance to defend their bodies and their property.” The counter-argument, of course, is that making guns more easily available is likely to lead to increased levels of injuries and fatalities. The 2011 incident, a fatal accident at a party, would seem to lend support to this view.

The debate in the academic literature has proved no less controversial. A 1997 paper in the American Journal of Economics and Sociology divided states into two groups: those with restrictions on gun ownership (such as permits and waiting periods) and those without. They found that although firearm deaths were slightly—and statistically significantly – less common in the states with restrictions than those without, to the tune of around 5 fewer deaths per 100,000 people, this difference dropped to a mere 3 per 100,000 after controlling for socioeconomic factors such as levels of poverty and alcohol consumption. (The math behind controlling for such factors is complex, but—in effect —involves matching states with and without gun restrictions on levels of poverty and alcohol consumption, and then comparing these states on the number of gun fatalities). The authors concluded that “states with gun control laws had almost 3 fewer deaths per 100,000 than states without any such laws”, adding—almost as an afterthought—“The relationship, however, is not statistically significant”.

This drew a swift, damning and unambiguously-titled response: No, your evidence doesn’t prove what you think it does! (exclamation mark and all). Tomislav Kovandzic (then of Florida State University, now of the University of Texas at Dallas) pointed out that a “difference” that is not statistically significant (in this case, of 3 deaths per 100,000) is, in fact, no evidence of a difference at all.

But just what is statistical significance anyway? Well, suppose that you randomly put together two groups of 100,000 people from across the whole of the US, and looked to see how many were killed by firearms. It’s actually pretty unlikely that the number would be absolutely identical for the two groups. It’s far more likely that the number of firearm deaths in each group of 100,000 would differ to the tune of two or three deaths by pure chance alone. Simplifying somewhat, what a test for statistical significance does is look to see whether the observed difference between the two groups (in this case, 3 fewer deaths in the states with than without gun control) is bigger than the difference that could turn up between two groups of this size by pure chance alone. And, in this case, it isn’t.

The authors of the original piece hit back, pointing out that they had only ever claimed that gun control laws have “a very mild effect”, and that their point was really that poverty seems to be a far more important factor in fatal shootings than the existence or otherwise of gun control laws. However, this isn’t the end of the story. In 2005, the lead author of the original study, Ik-Whan Kwon of St Louis University, published a new and more fine-grained analysis, rating states on six aspects of their gun control laws (registration, training, regulation of sales, storage, licensing and litigation) rather than simply dividing states into those with and without restrictions. Although socioeconomic factors were again important, this time, gun control was found to have a statistically significant relationship with a lower level of fatal shootings. That said, the effect was again small, ranging from 1 to 6 deaths per 100,000.

The debate continues to this day. Kovandic’s website lists several recent studies which suggest that stricter gun control laws have little effect on rates of gun homicide, or possibly even a detrimental effect (the argument for arming the law-abiding majority).

So who is right? Well, with apologies for the fence-sitting, the only thing that is clear is that neither the gun-control advocates or the cold-dead-hands conservatives can legitimately claim that the data are unambiguously on their side. The problem is that, as we saw in this column a few weeks ago with regard to the safety of home births, double-blind randomized control trials are all but impossible. This would involve finding two states matched for all relevant socioeconomic, criminal and gun-ownership measures (itself an almost-impossible task), and then implementing gun control laws in one but not the other. Strictly speaking, the state that does not get the new gun control laws should – as the “control” group – get some other superficially-similar intervention. And, ideally, neither the inhabitants nor the researchers should know which state is which (the “double-blind” part). Good luck designing that study!

In the absence of such studies, researchers will have to do battle over observational data, which can always be analyzed in a myriad of different ways. So let’s spare a thought for those poor Florida State Senators as they consider the concealed-carry-on-campus proposal: Whoever they call to give evidence, they are never going to find – if you will forgive the pun – a smoking gun.