Statistical Significance vs. Importance

All too often I have seen people use the words, statistically significant, and statistically important, as interchangeable. They are not. They relate to two, distinct measurements. Statistical significance is a measure of certainty as it relates to random variation. That is, is it unlikely that the event could have risen through random variation? While importance is a measure of the effects of the event. For example, we could measure two, five-pound weights enough times to show that they are statistically different. But the difference would not necessarily be important. On the other hand if in an experiment we found one death in a thousand, it would be considered not statistically significant, but it could be very important (if the treatment in the experiment was the cause of the death).

Statisticians have used the word significant to mean "unlikely to have occurred by chance alone," and they have used not significant to mean "not proven beyond a reasonable doubt." In this context, statistically significant differs on grounds for conclusions, while a non-significant result means the jury is still out. Not Significant does not mean Non-Existent. Absence of proof is not proof of absence.

Statistical significance may be unrelated to practical importance. All real differences, no matter how small, are statistically significant on a sufficiently large base. The larger the base, the smaller the difference that will be declared statistically significant.

When looking at the results of a research project, I like to ask myself the following questions:

Is it significant and important? Is it significant and not important? Is it non-significant, but important? Is it non-significant and not important?

Following are a few quotes I have found useful:

"Significance does not measure difference vs. no difference." "A difference to be a difference, must make a difference." "Significance measures the degree of difference." "The purpose of computing is insight, not numbers." "All things in nature are different." "All things in nature can be shown to be significantly different, given enough time, effort and money."