Suggest HN: Studies based on small sample sizes shouldn't be on HN

31 points

9 years ago

Reading about studies based on small sample sizes has in total misinformed me. I've been convinced of many surprising things based on a trust that the research was well conducted, and that published results are on average true. The building replication crisis has shown that this faith was misplaced, particularly in certain fields.

For instance, it's now clear that I would have believed more true things and fewer false things if I had not heard about a single result from social psychology. I'm still rooting out beliefs that I came to based on bad research. (Unfortunately my mind doesn't maintain an index of beliefs by source. I wish I could request this feature.)

I'm rightly bitter about this, and I see the problem continuing. For instance, this submission is worse than worthless: https://news.ycombinator.com/item?id=13144483 . It's impossible to tell what experiment was done without buying the paper, and on average experiments of this type have been shown to be of negative value: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ .

We're all worse off for reading and talking about these articles. I'd like to see HN add a heavy link penalty on all reports of research based on small samples, with an exception for good replication studies. Off the top of my head, n<10,000 might be a good threshold.

26 comments