ReplicationWiki

[J.Höffler]

We constructed a wiki website for replication. It lists more than 2000 empirical studies and information about replication material: code and data availability, software used, methods, origin and type of data, and if replications have been published and which results they had. It is a wiki, so everyone can contribute. One can discuss the studies and vote (anonymously) which studies should be replicated. So far the focus has been on some leading journals in economics, but it can easily be expanded to other fields. More than 700 studies are listed for which Stata was used, so it serves as a repository to search for code written by experienced researchers. It has already been used for several replication seminars internationally - searching for examples of code and data used for publications that employed methods that are taught in courses has proven very helpful.
Please join!

[Konrad Zdeb]

J.,

It's definietly interesting project. Coursera is currently running a course on Reproducible Research. It may be worth to mention this resource there as it's very on the lines of what the course is about.

[J.Höffler]

The project is now linked with RePEc, so for each study listed that also has a link to a RePEc page the ideas/RePEc page links back.

[Richard Williams]

This looks like a fantastic resource. I often want to try different methods with published research (e.g. would a different type of ordinal model produce superior results?) but it can be very hard to get the data and code to do so.

This is probably in the instructions somewhere, but can you further restrict the search? I'd like to take your Stata link, and limit the search to cases where both data and code are available (or at least data). I might also want to restrict it to, say, papers that used logit or ologit as the method. Maybe these things could be added to the other options -- or maybe they are there and I just don't understand it yet.

Hopefully you will see a lot of Sociology papers eventually get added. I would also think that there would be many Stata Journal articles that could be added, as replicability is generally a requirement for publication.

[J.Höffler]

Sorry for the late reply, I thought I get an email if there is one.
It is possible to restrict the search to studies that used Stata and have code and data in the journal archive: http://tinyurl.com/Statawdata-code
If you set the values to 2 instead of 1 you get those studies for which the code is available online but elsewhere than in a journal archive.

You can search for methods by adding a search string for methods used, in this case "ordered logit":
http://tinyurl.com/ologit
This so far only gives two results since this field has not been filled with much detail for many studies. We are currently discussing to introduce a more intuitive search in cooperation with a software company.

Adding Sociology studies or those of the Stata journal would be great. We could introduce categories for each discipline. Question that has to be solved is how to get the resources to do this. Original authors and journals could use the wiki to promote their studies but so far few from outside our team have contributed. I guess we need to raise more attention to the project until a sizeable crowd of users has been built. Having student assistants add studies does not cost much but still some funding is needed. Do you have suggestions on this?

[J.Höffler]

By now we have 860 studies for which Stata was used. Newly introduced features allow to easily search for software and methods used.

[J.Höffler]

Now it is also possible to browse by data types and JEL codes.

[J.Höffler]

YSI workshop with Richard Ball, Johannes Pfeifer, Edward Miguel, Jan H. Höffler, and Thomas Herndon

January 6 – 7, San Francisco, Mozilla Science Lab
The workshop will take place right after the Annual Meeting of the American Social Sciences Associations, which includes the Annual Meeting of the American Economic Association (AEA) at which the biggest event for job market candidate selection is held, on January 3-5, 2016. Experiencing this conference with job market can be a valuable experience for those who consider applying for an academic job after their studies. During the conference there will be a special session on Replication in Economics, also organized by those who organize the workshop, setup below.
The workshop will consist of mini-courses covering research transparency in empirical research and macro models that are neglected in the conventional economics curriculum. For young scholars it can be very useful to orient themselves by looking at how established researchers do their studies. By now there is a lot of material available but then it is often frustrating when one wants to take a look at how their analyses were done just to see that it cannot so easily be redone. This workshops intends to help young scholars to find out how to replicate others’ studies and how to archive their own research for future use and for others.
The workshop also will feature student presentation sessions, which will give Ph.D. candidates the opportunity to present and discuss their research in a collaborative environment. Applicants shall enter the title and abstract in the registration form (deadline is November 1) and submit the complete version to the Institute no later than December 1, 2015. Moreover, during the joined lunch and dinner there will be ample time for social interaction with students and teachers.