I agree with you that institutionalized replication can democratize information access about failed replications and thereby forward science because less money and time are waisted pursuing research ideas that others already know won't pan out.
From my own experience in economics research, however, I have one caveat to add on how replication actually might harm great research. I experienced quite often that people talked about a single study that failed to replicate in a single replication study, "ah but that doesn't replicate", as an argument to why one shouldn't do research in that direction. But that's not Bayesian reasoning! I wish there was a more easy/automatic way of communicating the posterior after a failed replication, instead of heuristically ditching an entire new field because of one failed replication...
This is a great point! I think in order for abundant replication of this kind to naturally occur, a somewhat widespread perspective shift on what a failed replication implies would have to occur as well. I’d imagine a failed replication shouldn’t imply “never research this topic again,” but rather, “for decision makers in the private and public sector, perhaps don’t invent a product or invest millions into a policy entirely based off of the original findings of the target study.” In the academic space, I’d argue a single failed replication should arise curiosity and further exploration, unless the original study was found to be very poorly or facetiously done (and even then).
I agree with you that institutionalized replication can democratize information access about failed replications and thereby forward science because less money and time are waisted pursuing research ideas that others already know won't pan out.
From my own experience in economics research, however, I have one caveat to add on how replication actually might harm great research. I experienced quite often that people talked about a single study that failed to replicate in a single replication study, "ah but that doesn't replicate", as an argument to why one shouldn't do research in that direction. But that's not Bayesian reasoning! I wish there was a more easy/automatic way of communicating the posterior after a failed replication, instead of heuristically ditching an entire new field because of one failed replication...
This is a great point! I think in order for abundant replication of this kind to naturally occur, a somewhat widespread perspective shift on what a failed replication implies would have to occur as well. I’d imagine a failed replication shouldn’t imply “never research this topic again,” but rather, “for decision makers in the private and public sector, perhaps don’t invent a product or invest millions into a policy entirely based off of the original findings of the target study.” In the academic space, I’d argue a single failed replication should arise curiosity and further exploration, unless the original study was found to be very poorly or facetiously done (and even then).