Get credible! ... Or a modest proposal to implement pre-registration in think tank research

[Note: This post first appeared at On Think Tanks. It was written by Aaron Erlich Assistant Professor of Political Science and Founding Member of the Centre for Social and Cultural Data Science at McGill University and Dustin Gilbreath a Policy Analyst at CRRC-Georgia. The Caucasus Research Resource Centers in collaboration with Aaron Erlich and Caucasus Survey recently announced a pre-registration competition for articles that will use the 2017 Caucasus Barometer Survey. This post is a reflection on how and why think tanks can and should use pre-registration based on the experience of setting up the competition and a summer workshop on pre-registration hosted at CRRC-Georgia in summer, 2017.]

It’s almost a cliche to say that think tanks operate on the basis of credibility. The media, politicians, and some in the general public have increasingly questioned think tanks’ credibility in recent years, with think tanks and tankers becoming increasingly thought of as lobbyists under a different name. Think tanks are not the only ones experiencing a credibility problem. Social science outside the think tank world is also in the middle of a credibility crisis. This crisis stems from the lack of reproducibility of results, scandals related to data fabrication, reliance on small sample studies, and questionable data analysis practices in the search of statistical significance. In response to this crisis, one proposal that aims to ameliorate the situation is the pre-registration of studies. Pre-registration not only represents an opportunity for social science, but also for think tanks to increase the credibility of their work, lighten workloads, and increase donor independence.

What is pre-registration?
A pre-registered study is one where research design elements like sample size, hypotheses, any experimental protocols, and statistical analyses are defined, justified, and placed in a secure registry prior to actually carrying out data analysis. Usually, this means registering the study prior to data collection. However, in some cases one can pre-register a design while data collection is ongoing or before the data are available to the researcher.

For example, the Caucasus Research Resource Centers (Aaron’s former employer and Dustin’s current) is currently holding a competition for papers on foreign policy preferences in the South Caucasus based on the 2017 Caucasus Barometer survey. The survey, at the time of writing, is entering the field and is expected to be complete at the end of October. However, the data itself will not be released until December. To participate in the competition, researchers must register their research design at the Open Science Foundation registry (one of a number of reputable locations to register a study) and then submit their paper based off their pre-registration (without results) to the journal Caucasus Survey. The papers will be reviewed and accepted or rejected without the results of analyses, hence taking away the incentive to find statistical significance.

Like in other studies, analysis commences once an organization has collected data. However, in contrast to an unregistered study, after data collection a researcher need only focus on carrying out the analysis described in their pre-registration plan (or even simply run pre-written code for analysis) and insert the tables and graphs into their report. The bulk of the registration document serves as the report, hence front-loading the writing. Any exploratory data analysis, not described in the pre-registration, is reported as such in the final report.

Why would a think tank pre-register a study?
From the perspective of a think tank, there are reputational advantages as well as more subtle bonuses for managing a think tanks’ workload. The most important advantage of pre-registration will likely be that it increases the credibility of the think tanks’ findings. While in the past, research consumers often simply considered quantitative work robust, today issues surrounding replicability, and statistical modeling like hacking data for statistical significance at the holy 5% level have cast a large shadow over a great deal of quantitative work. Pre-registration precludes statistical hacking among other issues like researcher degrees of freedom, thus leaving fewer avenues of attack for potential critics.

Besides increasing credibility, pre-registration of research design can be particularly valuable for think tanks who work on commissioned studies. While philanthropy is one of the main sources of funding in the United States, in the developing world, think tanks often survive on service contracts and grants for studies on specific issues. Pre-registration is beneficial for both the academic reasons outlined above as well as for the think tank’s work load and independence.

When it comes to workload, a pre-registered study shifts a great deal of effort to project start up, but has the potential to decrease the ultimate workload. Because hypotheses are specified ahead of time, researchers will have to start writing out their expectations instead of focusing solely on design at the start of projects. This means that clients need to agree beforehand on what analyses the think tank will perform, and that agreement can be put into the deliverables of any contract. In this manner, donors who contract research could be constrained in their ability to request more analyses at the end of the project (at least without paying and specifying that these were not pre-registered results). In turn, this prevents researchers from needing to run (potentially hundreds) of additional analyses at the end of a project, when the client is unsatisfied with the results for whatever reason or curious about some other result they had not thought of ahead of time.

When it comes to independence, in the current environment, many donors do not hesitate to pressure researchers to produce results supportive of donors’ positions. With a pre-registered study, researchers have listed out the exact analysis they will implement beforehand along with their expectations about the results of the analysis. Donors who have have been educated in the way this process works can use pre-registration to make stronger arguments. Moreover, the analysis can be built into the deliverables of the project. Hence, donors will be less facilely able to suggest a different analysis or measure in place of the one the researcher chose at the start, thus decreasing the number of avenues through which donors can apply pressure, particularly since changing the analysis would have cost implications.

Many in the think tank community might be skeptical of this proposal. Afterall, donors hold the purse strings. Still, we think that the process will benefit many donors. In many cases, donors want the highest quality study possible, even when motivated by short term goals, because it will help them advance their agenda. While if the donor is so baldly seeking a single result and not willing to accept anything else, we accept the fact that the proposal won’t work. In many case, however, we think donors would take enhanced quality at no cost in exchange for a loss of some control over the final product.

Pre-registration does have limitations and drawbacks. For example, it only works when a researcher can credibly demonstrate that they do not have access to data before pre-registering. Moreover, although it is likely the model could be applied to qualitative research in some form, to date, the model has yet to be implemented widely. Since qualitative research arguably comprises the majority of think tank research, the scope of use is somewhat limited.

While pre-registration is not panacea to the problems of social science or the problems think tanks face, it is a tool for think tanks to consider, which can enhance credibility and potentially decrease workloads and increase independence - three things we don’t think many tankers would be against.

Dustin Gilbreath is a Policy Analyst at CRRC-Georgia and the Communications Manager at Transparify. 

Aaron Erlich is an Assistant Professor of Political Science and Founding Member of the Center for Social and Computations Data Science at McGill University, and previously a Research Consultant at the Caucasus Research Resource Centers in the Tbilisi office.