An article about computational science in a scientific publication is not the scholarship itself it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures. – Buckheit and Donoho 1995
In 2003 Bruce Edmonds and David Halescalled their paper ‘Replication, Replication and Replication: Some Hard Lessons from Model Alignment‘ expressing both the necessity of replicating computational models and the little appreciated but significant effort that goes into such studies.
In our field replication usually corresponds to re-writing the simulation’s code. It is not an easy task because algorithms and details of implementations are particularly difficult to communicate and even if the code is made available simply copying it would be pointless. Equally, publishing one’s replication is not straight forward as, again, the language of communication is primarily the code.
The ReScience Journal is a brand new (just over one month old) journal dedicated to publishing replication studies. What sets it apart is that it is github based! Yes, you’ve heard it – it is a journal that is (almost) entirely a code depository. This simplifies the whole process and helps with the issue of ‘failed replications’ (when the replication rather than the original study has a bug). You upload your replication code and other researchers can simply fork out their implementations. How come nobody thought of this earlier?