research-- by the letter

From Carl Bialik-- the "Numbers Guy" at the WSJ (hat tip: Linda Christiansen)...

What people don't know about statistics-- laypeople and sadly, researchers-- can cause all sorts of odd inferences. For example, toward the end, Bialik draws a vital distinction between statistical "significance" and statistically meaningful.

We like our names. And that preference can have negative repercussions, according to research published last month. Major leaguers with "K" initials tend to strike out more, perhaps reflecting the batters' unconscious pull to appear next to the strikeout symbol "K" on scorecards. Students with initials C and D have worse grades than the A's and B's and everyone else, gravitating toward the grades their initials represent.

These findings received widespread coverage for their headline-friendly implications. Less apparent are the statistical parables. Several statisticians commended the authors for elegant work. But the study also demonstrates how pliable numbers can be, the surprising pitfalls of working with very large data sets, and how defensible yet debatable choices can influence final results....

The study is the latest to seek the factors that shape our destiny. The order in which children arrive in a family may determine intelligence and personality traits; the latest study, from Norway, found that first-born military conscripts have an IQ, on average, 2.3 points greater than second-born children. A 2005 book, covered on CNN and elsewhere, associated drivers' astrological signs with their accident rates -- watch out for merging Libras.

But these findings were hardly conclusive. Many contradictory studies have called into question the birth-order hypothesis. And the "Car Carma" research was based on a single set of about 100,000 drivers in a single year. There is no evidence that Libras have been relatively hazardous road companions so far this year....

Duke statistician Jerry Reiter said he'd also like to see whether Gregs and Garys are especially likely to ground out, or whether Edwards and Elmers commit more than their share of errors. "One would expect these to hold as well if the authors' hypothesis is right," Prof. Reiter says. "If not, the authors might be finding associations that do not have causal interpretations."

The authors are on firmer statistical ground elsewhere in the study. The association of slightly lower grades with the initials C and D, compared with all other initials, was highly significant statistically for a set of data covering 15,000 graduating business-school students. (Students with C and D names wouldn't be expected to perform worse in countries where those letters didn't correspond to mediocre grades, but we can't be sure because those numbers haven't been studied.)

University of California, Irvine, statistician Hal Stern points out something most media missed. The effect is tiny: 0.02 of a grade-point average point lower for the initials C and D (and this columnist isn't including that because of his first initial). Therein lies a lesson in the difference between statistical significance -- the confidence that there is some association between two factors -- and the strength of that association.

"In very large samples like the ones here, even small differences will be judged statistically significant," Prof. Stern says. "This means that we're confident the difference is not zero. It does not mean the difference we see is important." Prof. Nelson agrees that this effect is "so small that you shouldn't worry about it" when naming a child, though he does say the study exposes an example of how the unconscious mind can undermine conscious motivation.

But Bowling Green statistician Jim Albert warns: "You can prove any silly hypothesis ... by running a statistical test on tons of data."