Statistics don’t lie—but they can be misleading

When it comes to deciding how significant a statistic is, Université de Montréal education professor Sébastien Béland is clear: “Our goal is to change practices in order to improve quality in the social sciences and humanities,” he said. 

The standard measure of statistical signficance is the p-value, which describes the statistical probability of obtaining the observed results. But this is a number that is often poorly understood—not only by the media and the general public but also by the academic community. 

“Much science of all kinds—the social sciences, the humanities, health sciences—relies on the p-value,” Béland noted. “But for years, researchers have been urging caution. We can’t say something is or isn’t a discovery on this basis alone.” 

The flurry of research during the COVID-19 pandemic, and the intense media attention given to the findings, had some unfortunate results, he said. “Things went sideways in many cases. Results were misinterpreted and it led to poor public policy.” 

That's when he really became interested in the p-value.  

In June, with Michael Cantinotti of Université du Québec à Trois-Rivières and Denis Cousineau of the University of Ottawa, Béland published an article on the subject in the journal Canadian Psychology/Psychologie canadienne. 

Twenty-four co-signatories from various university faculties and departments added their voices, including several from UdeM (Christian Bourassa and Christophe Chénier, education; Stéphanie Forté and Quoc Dinh Nguyen, medicine; Éric Lacourse, sociology; Floris Van Vugt, psychology). 

“We wanted to take a stand and make an impact,” Béland said. 

And while most scholarly articles on methodology are in English, this article is in French. “It’s important to publish in French,” Béland observed. “It has been shown that it’s easier to learn a complicated subject in your mother tongue.”

Should the p-value be abandoned in the social sciences? Not at all, he and his co-authors say. What's needed is simply more nuance. “The p-value doesn’t tell the whole story; it’s part of the scientific picture,” Béland argued. 

He believes scientists in disciplines other than statistics should think more critically about methodology: “It’s become routine to draw the line at 0.05. But results above that line can also be meaningful.”  

Béland plans to use this article in his classes to make his students aware of the issue. He is also thinking of writing a book on how to read and report statistics, as they are often misunderstood and misreported in the media.  

“Journalists have little training in this area, and scientists aren’t always helpful,” he noted. “It can lead to overblown interpretations.” 

In the end, “I’m not saying we should abandon the p-value, but we have to keep it in perspective,” he said. “It’s important, but it isn’t always the final word.”