The HitchHARKer’s Guide to the Significance

In contribution to the overall reigning force of “publish or perish” we want our study results to be positive and significant, no matter what. There are many ways in increasing the probabilities of getting a significant result and therefore a publication in renowned magazines i.e. our next paycheck! In the following, I will illustrate disguised techniques in order to help frustrated researchers with getting their next 15 minutes of fame on the research question of whether morning coffee increases productivity or not (not is not an option). In other words, we want to produce a false positive by misusing our degrees of freedom in designing and working on our study (so-called “p-hacking”; Simmons et al., 2011).

Several steps to the truth - you won't believe step 4! 

The first step is to not preregister the study. In skipping this step, we are building up a wall for either scientists and non-scientists who are trying to replicate or understand our methods of research and our thinking. After the first step is done, we can open the door to our secret lab and start with the real work: subjectivity. When thinking about a research question like “does morning coffee improves productivity” it is important to stay as vague and subjective as possible. Productivity can be defined and measured in many ways and “morning coffee” does not imply the amount or the caffeine level of the coffee. With measuring things like caffeine level, time of the coffee input, amount of coffee over the day, the brand of coffee, brewing method, we have a robust basis for choosing the “right” IV. This procedure is also applicable to our DV, the productivity. Productivity can be measured as the amount of work done in one hour, a day, a week or a month, but it could also be measured in counting toilet break times or social interactions with co-workers. In addition to the IV and DV, we could also include or not include covariates, depending on whether they strengthen our assumed connection or not. The randomness and flexibility in the use of covariates is also a point which is seldom explained in many studies and therefore adds to the hideous intransparency. This flexibility in analyzing dependent variables almost doubles our chances in finding a false-positive result (Simmons et al., 2011).

Rather tweak than weak - c00l h4ck1n6 

When we decide in using an experimental design with two groups, we could tweak our results in either doing a lot of experiments with different conditions supporting our hypothesis or just repeating the one experiment multiple times, until we reach one significant result. Thanks to our friend the alpha level, on which we can also decide freely, but for trustworthiness, we should use the common alpha level of α = .05, we only have to repeat our experiment at least 20 times. This is due to the given probability of falsely rejecting the null hypothesis of 5%. So, even if the connection between our variables is not significant, we still have a small chance to produce a significant result. 

If we don’t want to do this work, we could also change our “wrong” hypothesis into a hypothesis that fits our first run of experiments. This method is called HARKing (Kerr, 1998) and should also not be reported. A good and strong HARKer should also always be vigilant about “researchers” trying to replicate his study and negating the one true hypothesis. Therefore, it is necessary to delete evidence of the prior hypothesis and to only report the “after-data-hypothesis”. Even though a lot of literature is seeing HARKing as a threat to good science (Kerr, 1998), it is not prohibited and within the rules of publication. 

Last but not least, we could also p-hack our data on our way into significance. This is done for example via the deletion of outliers, or the creating of groups or subgroups (low caffeine, medium caffeine, high caffeine). Speaking of groups, the creation and definition of a sample size is also a good way to support your hypothesis in whichever way you can possibly imagine. First of all, we should only add subjects to our study as long as there is no significant effect. This method increases our chances in finding the (false) positive result at a rate of 50% (Simmons et al., 2011). In addition, choosing a large sample size endangers the chance to find a false positive result critically (Simmons et al., 2011).

Sincere hopes for better science - and a better world

With this article, I am happy to contribute to a better (more truth!) world, and hope to encourage scientists to do the things they love in an easy and replicable (urgh, silly word) painless way. Pain is considered as not good and a little truth doesn't hurt anyone, right? The next articles will be named "Changing the Change - how to change the climate in three easy steps", "Give me back my abbreviation - a personal statement from a personal device", and "Accomplishment as a sense - how to make even more money than in science". They will all be true, trust me.

Some references

Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22, 1359-1366.