One of the reasons I started a blog was that one can not fit some ideas in a single tweet.
Background
So, recently D. Lakens (whose blog I also recommend) shared (on Twitter) his conceptual analysis about what preregistration does and what it is for (and what are the nice things it is not for but does anyway, i.e. its positive externalities). Tweets below:
The paper linked therein[1] is easily available (pdf) and short and has interesting concent, so I recommended reading it. For example, I like the idea of severe testing as important part in doing science (one of the reasons I like the idea of preregistration, too!) For example, in addition to quoted stuff, there is a good thought experiment[2] that demonstrates how judging the severity of a test in preregistration is not as clear-cut matter as it first may seem.
Comment that didn’t fit Twitter character limit
Briefly stated: I am not convinced it is so easy to resolve what positive effect is and what is not a positive externality in this case, and (more practically) whether the difference is important one. edit. And if it is important, maybe that means someone should do something to address them, too!
Why? I have an example concerning the positive effects from another scientific practices. I have personally some opinions against the current scientific publication process, peer review and all, but recently read a point in favor of it; I forgot where, but I remember I could not easily dismiss it. The crux of the argument was as follows:
People write more readable and generally better papers, include certain results and method descriptions, go through STROBE etc checklists, and in many other ways carefully “dot their i’s” while writing drafts before submitting them to publication, because they know that those are the first things the whole peer review + publication process will complain about if they are missing. Anticipation of scrutiny results in improved outcomes.
In ideal world, they could do all those good things anyway without the process, but this being not the ideal world, they probably wouldn’t. Thus, current process is good and necessary. [/end of argument]
(Well, not necessary. My take is: any proposal to move away from current reviewing process should present an equivalently good mechanism for ensuring that those particular minimum remains, or better yet, should improve it. But back to positive externalities of preregistration.)
It seems to me that the same idea applies to the positive externalities of preregistration. There will be less “careful thinking about analyses before collecting data” if there is no mechanism where researchers reliably are required to do careful thinking. There will be fewer questions that amount to “what would falsify your claims” asked if there is no mechanism where researchers reliably face such questions.
So, is preregistration good and necessary …. ? Here I see the conclusion can go two ways: If preregistration is the only tool around that reliably enforces the “positive externalities”, maybe the externalities are not practical externalities, and yes, their existence would be a good argument in favor of preregistration. On the other hand, if the mechanism to produce those positive externalities can be decoupled from preregistration, it should be of possibly high importance to get an initiative (similar to pro-preregistration efforts) going on for disseminating the decoupled mechanisms. “Possibly high”, as their importance should probably be proportional to how important (or unimportant) the positive externalities are viewed as an argument for preregistration.
There is also a third possibility: The externalities-or-not are not positive enough to warrant costs or harmful effects of such efforts. I would think such costs can be made low. At minimum, if the method is simpler then preregistration process, it would be more cost-effective than preregistration. At worst, it would look about the same as current proposals for preregistration.
Maybe the decoupled method could be a simpler an internet service similarly to we currently have for preregistration, but with shorter and less restrictive checklists. Maybe it could be some sort of standard evaluation criteria that should be formally incorporated in the review process. Not sure, but sounds like an idea?
Notes
[1] Lakens, Daniël. “The Value of Preregistration for Psychological Science: A Conceptual Analysis.” Japanese Psychological Review 62, no. 3 (2019): 221–30. https://doi.org/10.24602/sjpr.62.3_221.
[2] Paraphrasing: Suppose researcher A preregisters a test for linear model, which leads to nothing, but finds out in explorative phase there appears to be a polynomial effect. As researcher A did not preregister polynomial model, they argue the polynomial test was less severe. However, researcher B finds a polynomial model a more natural to test a priori and would have preregistered it if they had designed the study. Which test is more severe?
Aaro Salosensaari - Homepage 