This just in: "Tall Women at Greater Risk for cancer!"  This just in: "Coffee May Lower Suicide Risk!"  This just in: "Living Near Industrial Plant Raises Cancer Risk!"

Everyone has seen these headlines and hope that drinking coffee in the park will make us taller.  The first thing I would like to point out is that we seem to be bombarded with information at every turn about what makes life riskier, as though if we make enough of the right choices we would eliminate risk.  That of course is not possible and the constant stream of information about risk, and especially risk without perspective, serves only to give us a feeling of helplessness mixed with dread.

The other problem is that of the trivial finding in research.  The trivial finding is actually not so trivial because it tends to cloud the larger issue and lead researchers, and readers, down ill-considered paths.

What is a trivial finding?  There are a couple of definitions; one would be a relationship or difference that is so small that it is meaningless in the larger context.  Take for instance height and cancer.  Yes, the relationship is there, but it is all but washed out by other risk factors such as smoking, age, and family history.  You could be 4 feet tall, low risk if for cancer by the height standard, but if you smoke 3 packs a day and had a mother die of breast cancer your height is irrelevant. The second type of trivial finding, and often related to the first, is the result of data mining.  If you measure enough variables on enough subjects then the changes of finding either spurious relationships, or extremely weak relationships, increases.  Finding spurious relationships is a type I error.  Finding very weak relationships or differences is in some ways a flaw of high statistical power.  If you survey 100,000 K-12 students on 50 nonsense variables chances are you will find some relationship simply because of the number comparisons or relationships you explore.  Remember that the probability of type I error accumulates with each comparison.  Even sophisticated statistical methods such as logistic regression and factor analysis are not immune to these effects, although less so that simpler methods such as the correlation matrix or ANOVA.  There is also the problem of numerous variables often measuring similar constructs and inadvertently forming relationships with each other

How do you avoid falling into the trivial trap?  The first and most important method is a solid theoretical structure to the research.  Fishing trips will almost always catch something, including an old boot.  You need to hunt for a specific target with a specific theory and hypothesis.  This way when small differences or relationships are found they can be considered in light of the overarching questions being asked.  Trivial findings can then be ignored.

This leads to the second way to avoid the trivial trap.  Ask the question "does it make sense?", and not just in the "can I make up some sort of argument to get it published" sense.  It must fit into that larger theoretical framework or it should be regarded as trivial.  This is a question that can be asked not only of your own research, but also research that you read in news reports and in journals.

The third way to avoid the trivial trap is to ask, "What are the actual numbers?"  Reports love to state relative risk.  I was at a conference many years back and one presenter remarked that exposure to a certain chemical doubled the mutation rate of cultured cells.  When I asked what the actual numbers where the researcher revealed that the mutation rate increased from 3 cells/100,000 to 6/100,000, in essence a trivial finding.  (It also showed on his face that he knew he had been caught).  This question also gets at how much does it matter in light of the other important variables that contribute to the question at hand. 

I am not arguing that small differences and relationships do not matter.  They can.  In the public health arena a 5% increase in infant survival is important both in terms of numbers because of the thousands of births that occur each year, but also in a moral sense.  However, distinguishing small differences requires extra scrutiny.  Perhaps small differences need a more rigorous test of significance.  Even Ronald Fischer, one of the founders of modern statistics, believed that setting alpha to .05 was arbitrary and only a starting point.  He believed that in many instances a more rigorous alpha level, combined with theoretical scrutiny, was needed before determining that a finding was significant.

So, where does that leave us?  My take home message is that the statistical method cannot stand alone as the only judgment of any research finding.  It must be married to the greater theoretical question at hand and careful intellectual analysis.