Member-only story
The anatomy of a bad idea
I had a bad idea.
But I won’t tell you about a specific bad idea — I will tell you about all of them.
I will generalize to a pattern that many ideas — bad and good — follow. I’m hoping that you — yes, you — will find this exercise helpful or interesting. If not, well, it wouldn’t be the first bad idea I’ve had.
What’s an idea?
By “idea,” I am implicitly talking about solutions to problems. For me, those are usually abstract concepts that require mathematical modeling to solve technical problems.
A secondary motivation for writing this is that people — and, importantly, that includes scientists — should be more open about their bad ideas and failed experiments. But before all this starts to sound too negative, I want to emphasize that “bad” is really good!
Negative results
People rarely like to admit failure. It’s no surprise, really. There are a myriad of psychological and social reasons not to. On the other hand, with maxims like “fail fast, fail often,” it is clear that failure is a prerequisite for success. Of course, it’s not failure per se but how one fails and iterates that determines eventual success.
In science, a study that fails is one that does not find a connection between experimental variables. These are called negative or null results. In most academic journals, negative results are not publishable. This is related to the infamous p-value.
Roughly speaking, the p-value of a result is the probability of obtaining the result due to chance. Smaller is obviously better, but the exact statistical interpretation is subtle and the subject of a lot of controversies over the past decade spurred on by the so-called reproducibility crisis.
Until recently, in medical trials and social science, a p-value of 0.05 was defined to be “statistically significant.” Many have suggested that this arbitrary threshold is not stringent enough, and 0.005 is more appropriate. In high-energy particle physics, the threshold to report “evidence” of a particle is a p-value of 0.003, and to report a “discovery” is a p-value of 0.0000003. Others advocate moving beyond p-value altogether.
