Getting at the truth behind the numbers

May 31 2005
The Chronicle (Ladysmith) EDITORIAL:

Edward Hill's story this week on pollution data brings to mind a little book written in the 1950s called 'How to Lie with Statistics'. This book points out many of the pitfalls of scientific research we as consumers of information often take as the gospel truth without looking a little more closely at the numbers.
In our story on page 15, a report from the Commission for Environmental Cooperation states the Crofton mill released 1.4 million more kilograms of toxins in 2002 than in 1995, giving the mill the dubious distinction of having the largest pollutant increase in Canada during this period, and the fourth largest increase from 1998 to 2002.
On the face of it these numbers make the Crofton mill look pretty bad, but in fact, as the story points out, there are a couple of problems with this study. The first is the study does not tell us who is the worst polluter, it just tells us who had the biggest increase in pollutants. This is like comparing two hockey players and awarding Player A for going from five goals a season to 40 (a 700 per cent increase), versus another player who went from 40 goals to 50 (a 25 per cent increase). Player A showed more improvement, but Player B was clearly the better player because he started off with a higher goal-scoring percentage. Similarly, the pollution study doesn't give us a baseline of how bad these mills were to begin with.
The study also lacks credibility because the data used to compare emissions was drastically different from one year to the next due to Norske's system of reporting. In 1998, for instance, the mill only reported two compounds, but in 2002 it reported 23. Betweeen 2002 and 2003 the mill emitted over 300 more tonnes of methanol, but due to changes in mathematical emission factors, it's not clear whether more, less, or the same amounts of methanol left the stacks.
From this data, therefore, we can't draw the conclusion that Norske is a better or worse polluter, only that the company is now reporting more emissions than it did in the past. As mill manager Don McKendrick admitted, "the compounds were always there, just not reported on."
Cold comfort, no doubt, to the mill's opponents.
The media is often confronted with scientific reports such as these that paint certain groups in a negative light, and it up to us, as reporters and editors, to find "the truth" behind the numbers.
Another example is the annual Fraser Institute report card on schools, which ranks the province's elementary and secondary schools based on a set of criteria. Every year newspapers dutifully trot out articles about how high or low their local schools ranked on the list, even though this study has been reported to show a bias toward large urban schools. The report card does not take into account less quantifiable factors such as locally developed programs or the more close-knit learning environments often found in rural schools.
This doesn't mean we shouldn't report on the Fraser Institute findings or any other study that purportedly presents scientifically collected data, but it does mean we have a duty to examine the other side of the story so that people can draw their own conclusions as to the merit of the information presented.
- Andrew Topf, Editor