Get credible! ... Or a modest proposal to implement pre-registration in think tank research

[Note: This post first appeared at On Think Tanks. It was written by Aaron Erlich Assistant Professor of Political Science and Founding Member of the Centre for Social and Cultural Data Science at McGill University and Dustin Gilbreath a Policy Analyst at CRRC-Georgia. The Caucasus Research Resource Centers in collaboration with Aaron Erlich and Caucasus Survey recently announced a pre-registration competition for articles that will use the 2017 Caucasus Barometer Survey. This post is a reflection on how and why think tanks can and should use pre-registration based on the experience of setting up the competition and a summer workshop on pre-registration hosted at CRRC-Georgia in summer, 2017.]

It’s almost a cliche to say that think tanks operate on the basis of credibility. The media, politicians, and some in the general public have increasingly questioned think tanks’ credibility in recent years, with think tanks and tankers becoming increasingly thought of as lobbyists under a different name. Think tanks are not the only ones experiencing a credibility problem. Social science outside the think tank world is also in the middle of a credibility crisis. This crisis stems from the lack of reproducibility of results, scandals related to data fabrication, reliance on small sample studies, and questionable data analysis practices in the search of statistical significance. In response to this crisis, one proposal that aims to ameliorate the situation is the pre-registration of studies. Pre-registration not only represents an opportunity for social science, but also for think tanks to increase the credibility of their work, lighten workloads, and increase donor independence.

What is pre-registration?
A pre-registered study is one where research design elements like sample size, hypotheses, any experimental protocols, and statistical analyses are defined, justified, and placed in a secure registry prior to actually carrying out data analysis. Usually, this means registering the study prior to data collection. However, in some cases one can pre-register a design while data collection is ongoing or before the data are available to the researcher.

For example, the Caucasus Research Resource Centers (Aaron’s former employer and Dustin’s current) is currently holding a competition for papers on foreign policy preferences in the South Caucasus based on the 2017 Caucasus Barometer survey. The survey, at the time of writing, is entering the field and is expected to be complete at the end of October. However, the data itself will not be released until December. To participate in the competition, researchers must register their research design at the Open Science Foundation registry (one of a number of reputable locations to register a study) and then submit their paper based off their pre-registration (without results) to the journal Caucasus Survey. The papers will be reviewed and accepted or rejected without the results of analyses, hence taking away the incentive to find statistical significance.

Like in other studies, analysis commences once an organization has collected data. However, in contrast to an unregistered study, after data collection a researcher need only focus on carrying out the analysis described in their pre-registration plan (or even simply run pre-written code for analysis) and insert the tables and graphs into their report. The bulk of the registration document serves as the report, hence front-loading the writing. Any exploratory data analysis, not described in the pre-registration, is reported as such in the final report.

Why would a think tank pre-register a study?
From the perspective of a think tank, there are reputational advantages as well as more subtle bonuses for managing a think tanks’ workload. The most important advantage of pre-registration will likely be that it increases the credibility of the think tanks’ findings. While in the past, research consumers often simply considered quantitative work robust, today issues surrounding replicability, and statistical modeling like hacking data for statistical significance at the holy 5% level have cast a large shadow over a great deal of quantitative work. Pre-registration precludes statistical hacking among other issues like researcher degrees of freedom, thus leaving fewer avenues of attack for potential critics.

Besides increasing credibility, pre-registration of research design can be particularly valuable for think tanks who work on commissioned studies. While philanthropy is one of the main sources of funding in the United States, in the developing world, think tanks often survive on service contracts and grants for studies on specific issues. Pre-registration is beneficial for both the academic reasons outlined above as well as for the think tank’s work load and independence.

When it comes to workload, a pre-registered study shifts a great deal of effort to project start up, but has the potential to decrease the ultimate workload. Because hypotheses are specified ahead of time, researchers will have to start writing out their expectations instead of focusing solely on design at the start of projects. This means that clients need to agree beforehand on what analyses the think tank will perform, and that agreement can be put into the deliverables of any contract. In this manner, donors who contract research could be constrained in their ability to request more analyses at the end of the project (at least without paying and specifying that these were not pre-registered results). In turn, this prevents researchers from needing to run (potentially hundreds) of additional analyses at the end of a project, when the client is unsatisfied with the results for whatever reason or curious about some other result they had not thought of ahead of time.

When it comes to independence, in the current environment, many donors do not hesitate to pressure researchers to produce results supportive of donors’ positions. With a pre-registered study, researchers have listed out the exact analysis they will implement beforehand along with their expectations about the results of the analysis. Donors who have have been educated in the way this process works can use pre-registration to make stronger arguments. Moreover, the analysis can be built into the deliverables of the project. Hence, donors will be less facilely able to suggest a different analysis or measure in place of the one the researcher chose at the start, thus decreasing the number of avenues through which donors can apply pressure, particularly since changing the analysis would have cost implications.

Many in the think tank community might be skeptical of this proposal. Afterall, donors hold the purse strings. Still, we think that the process will benefit many donors. In many cases, donors want the highest quality study possible, even when motivated by short term goals, because it will help them advance their agenda. While if the donor is so baldly seeking a single result and not willing to accept anything else, we accept the fact that the proposal won’t work. In many case, however, we think donors would take enhanced quality at no cost in exchange for a loss of some control over the final product.

Pre-registration does have limitations and drawbacks. For example, it only works when a researcher can credibly demonstrate that they do not have access to data before pre-registering. Moreover, although it is likely the model could be applied to qualitative research in some form, to date, the model has yet to be implemented widely. Since qualitative research arguably comprises the majority of think tank research, the scope of use is somewhat limited.

While pre-registration is not panacea to the problems of social science or the problems think tanks face, it is a tool for think tanks to consider, which can enhance credibility and potentially decrease workloads and increase independence - three things we don’t think many tankers would be against.

Dustin Gilbreath is a Policy Analyst at CRRC-Georgia and the Communications Manager at Transparify. 

Aaron Erlich is an Assistant Professor of Political Science and Founding Member of the Center for Social and Computations Data Science at McGill University, and previously a Research Consultant at the Caucasus Research Resource Centers in the Tbilisi office.

Municipal Transparency Ratings

In 2014, CRRC-Georgia requested information from Georgian municipalities through the Ministry of Regional Development and Infrastructure (MRDI). Only 37 of 63 municipalities responded to our request. 31 of these provided some information. Only 17 provided complete information in response to our request. Based on this experience, CRRC-Georgia decided to rate the municipalities’ responses. While a score of 0 means that the municipality didn't respond at all, municipalities received 1 point for at least responding to the letter. Municipalities which received a score of 2 provided us with some of the information requested, while municipalities that scored 3 provided all requested information.
 

Months later, we repeated our endeavor. However, this time, freedom of information requests were submitted directly to the municipalities. This time around, we asked for data in an Excel file. Almost all the municipalities responded, however, the quality of responses varied. 44 municipalities sent Excel files and 39 contained the requested information. An analogous rating system was used to rate the responses to this round of freedom of information requests.
 

Based on these two rounds of our unintentional rating of municipal transparency, we created an index of municipal FOI transparency, summing up the two scores. Zero corresponds to no response for both rounds of requests while six reflects response with full data provided.

Perceptions of court proceeding transparency

[Note:  Social Science in the Caucasus is publishing the work of six young researchers who entered CRRC-Georgia’s Junior Fellowship Program (JFP) in February 2015. This is the sixth blog post in the series. Click here to see the first, second, third, fourth, and fifth posts in the series.]

By Mari Mekhrishvili

Transparent courts are essential to ensuring the accountability of the judiciary and to sustaining society’s confidence in the judicial system. The current version of the Organic Law of Georgia on Common Courts, after important amendments made in 2013, is designed to ensure the transparency of the courts. The courts are now obliged to record court sessions and provide records to all interested parties upon request. In addition, the Public Broadcaster is authorized to record and broadcast court sessions except in cases when sessions are closed either in part or in whole, and to provide records to other media outlets upon request. The law also guarantees that the prosecution, defense, and any person present at the trial can record court sessions. 

To find out to what extent the legislative changes about photo, video and audio recording during the court sessions works in practice, the Georgian Young Lawyers Association surveyed 26 Georgian media outlets. According to the results of this survey, court proceedings are truly transparent in Georgia, as all media requests to receive court session records were granted from the Public Broadcaster when carried out in compliance with law. 

In this context, this blog post looks at Georgian citizens’ perceptions of court transparency in 2014 using CRRC-Georgia’s 2014 Attitudes towards the Judicial System in Georgia survey, funded by East-West Management Institute and the United States Agency for International Development.

Respondents were asked whether, in their opinion, Georgian court proceedings were transparent (a) before and (b) after the milestone 2012 Parliamentary Elections. Only 13% reported that courts were transparent before 2012. This is the period, when only the courts had the authority to record and stenograph court sessions, which could still be banned by the judge’s “reasoned decision”. 34% reported the same for the period after 2012 elections.

Note: The answers to the question, “To what extent do you agree or disagree with the opinion that court proceeding are transparent in Georgia?” were re-coded from a 10-point scale used in the questionnaire into a 3-point scale, where original options 1 through 4 were combined into “Disagree,” options 5 and 6 were combined into “Neither agree nor disagree,” and options 7 through 10 were combined into “Agree.”

Differences in attitudes are evident in different settlement types. Only 7% of Tbilisi residents reported that the courts were transparent before 2012, compared with 13% of urban settlements besides the capital and 17% of rural residents. Perceptions of court proceeding transparency after the 2012 Parliamentary Elections followed a similar pattern with 20% of capital residents, 33% of residents of urban settlements besides the capital, and 44% of rural residents agreeing with the statement that court proceedings were transparent. Rural residents generally appear to believe most in transparency of courts, though, as it is well known, rural residents always report higher levels of trust in institutions.

Hence, people’s perceptions of court transparency in Georgia differ when assessing the situation before and after 2012 Parliamentary Elections, but in both cases the rural and urban populations have very different assessments. 

Take a look at the 2014 Attitudes towards the Judicial System in Georgia survey, here.

The CRRC’s 500th post and thoughts about the future of social research

By Hans Gutbrod

When we started this blog, quite a few years ago, we published a few posts, and didn't tell anyone about it. We weren't sure whether the venture would work – and whether it was a good fit for ourselves. Now, with this 500th post, I'm glad to see that our tentative venture took off.
The blog continues to hold a lot of material that may be of interest to readers, as it documents various aspects of work that have been done over the years. Readers, researchers, journalists and the interested public may find it useful to browse whenever they look for information on a specific issue.

An additional tool we built for that purpose is Find Policy, a tool that searches websites of all major Georgian research organizations, and also includes this blog. You can find it here: www.findpolicy.org/georgia

At the occasion of the 500th post, what are some of the issues that lie in the future? How can empirical social science be relevant to the lives of people in the South Caucasus? Here are some ideas.

Quality & Standards
Research organizations succeed because of processes. Being organized is often more important than being smart. If social researchers in the South Caucasus want to be credible, they have to deliver consistent quality, and this may sound difficult. However, there is much experience in how to structure review processes which institutions can draw on.

Such review costs time. Yet it’s an investment that pays off. If research organizations want to make a difference, putting such internal review processes in place is a key component. How to make them stick? Write them up and put them on your website, as a binding policy to commit everyone on the team. You can still override in rare emergencies, but you can’t thrive if you only operate in emergency mode.

Transparency
Fortunately, many organizations in Georgia are transparent about who funds them. In the past, these were primarily Western-oriented institutions. Lately, there is added concern about funding from Russia, seeking to further Russian interests. At first glance, this is legitimate, too, as debate typically thrives on a diversity of opinion. However, all institutions should be transparent about who funds them. They owe this to citizens, who thereby can better understand who paid for research. Organizations that are committed to having an impact should be role models in that regard. Georgia is already doing quite well – and why not become the most transparent country, in that regard, in the world? (Disclosure: I'm campaigning on this issue through an initiative called Transparify.)

Funding & Finance
Research organizations in the South Caucasus have enjoyed generous support. However, donors often do not understand what it takes to fund consistent quality. For that reason, too, transparency matters. Many research organizations in the South Caucasus arguably are better characterized as bundles of short-term projects. Such funding is valuable, but makes it harder to build up long-term expertise on particular issues. One worthwhile investment for donor money is to (continue to) give core-funding to those institutions that are genuinely committed to doing quality work. One evidence of such a commitment to quality are the standards and the transparency mentioned above.

Policy Relevance
What information should be in social research that aspires to be policy-relevant? There is no consensus on that in the South Caucasus, and arguably not even a sensible debate about some of the core components. Arguably, all policy proposals should be (a) intelligible, (b) calculate costs and benefits, on the basis of sound data, (c) consider alternatives and potential unintended consequences and (d) list major risks and corresponding mitigation strategies. These are some basic criteria that most people should be able to agree on, even while they add further suggestions.

In general, the quality of reports produced by research organizations in the South Caucasus has gone up, but I still read too many reports that fail by some of these basic standards. There also are presentations where I still struggle to understand what’s really being said. By the 1000th post, in šāʾ Allāh, it would be great if these basic problems were under control. Social research would increase its impact and help improve the lives of people in the South Caucasus.

With this happy anniversary note, I pass the ball over to you, readers, for your suggestions on what you would like to see from future social research in the region.

Hans Gutbrod was Regional Director at CRRC from 2006 to 2012, and initiated the CRRC Social Science in the Caucasus blog. He is now with Transparify. He is on Twitter. 

Trust and Distrust in Political institutions in Azerbaijan

[This is a guest blog post by Anar Valiyev, Azer Babayev, Hajar Huseynova and Khalida Jafarova, prepared in the framework of the Research Beyond the Ivory Tower project of the Norwegian Institute of International Affairs (NUPI). The views expressed in this blog are those of the authors alone, and may not reflect the views of CRRC-Armenia, CRRC-Azerbaijan, or CRRC-Georgia.]

This blog post is based on research on (dis)trust in political institutions in Azerbaijan. Internationally, levels of trust in political institutions often reflect how well these institutions perform in relation to citizens’ expectations. This is not necessarily the case in Azerbaijan. Although survey results consistently show that trust in the President is high, crucial political institutions such as Parliament and local government are less trusted and, overall, the level of trust is not very high. This is important because of two implications: on the one hand, high levels of political trust can aid economic development and political stability. Distrust in political institutions, on the other hand, can lead to low voter turnout and the weakening of democratic reforms. Allegedly, it may also lead to corruption.

As in other post-Communist states, in Azerbaijan, political distrust is historically rooted. Even during the Communist regime, trust toward state institutions was not high due to high levels of corruption, nepotism and mismanagement of institutions. Since the collapse of the Soviet Union, declining trust levels and poor public management have led to political apathy and low membership in political organizations.

In Azerbaijan, which is a presidential republic, trust in the President and trust in other institutions of the executive government differ from each other. While, according to CRRC’s Caucasus Barometer survey, 56% of the population reports trust and 22% distrust in executive government, e.g. the Ministers, 84% of the population reports trusting the President of Azerbaijan. However, only 39% of the population reports trusting local government. Attitudes towards local government are also reflected in very low turnout during municipal elections – 32% in 2009, 23% in 2011, 39% in 2014. It is possible that people do not see much value in participating in these elections due to the ineffectiveness of municipalities.

In order to address the issue of distrust in political institutions, the government should take immediate action by making the work of state institutions transparent and public, educating the public on the importance of state institutions, and involving the public in the decision-making processes.

E-transparency in Georgia: A key to faith in democracy?

Can more available public information on Georgian governmental websites promote institutional trust and enhance faith in democracy in Georgia? This is the topic of a recent book called “Electronic Transparency in Georgia” by the Institute for Freedom of Information (IDFI). The book presents findings on the accuracy and relevance of information provided on various Georgian governmental websites. The relevant research was carried out in Georgia between August and December 2010 and was within the wider framework of IDFI’s aim to promote access to public information, including budgets, staff salaries in different ministries, among other types of information.

Electronic transparency is becoming more of an issue as internet usage gradually increases in Georgia. The figure below shows that according to the Caucasus Barometer (CB), 18% of Georgians used the internet every day in 2010, whereas 13% used the internet every day in 2009. Data from the CB also consistently shows that those who more frequently use the internet are younger Georgians with more education and live in the capital.

IDFI observes some positive improvements to e-transparency such as the introduction of an online system in October 2010 by the State Procurement Agency. This online system will publish details about tenders and increase transparency in procurement procedures. However, according to IDFI, most governmental websites in Georgia do not display information about their functions, budgets or details about the implementation of state-funded projects. Some sites also do not feature a postal or email address. Subscriptions to news bulletins are rare, as are open platforms where people could express their opinions through forums, polls or blogs.

Can a lack of public information online influence faith in democracy in Georgia? This is especially relevant when bearing in mind the results of the survey,“Public Attitudes Towards Elections in Georgia”conducted by CRRC in April 2010, where 48% of Georgians said they did not believe that Georgia was a developed democracy at that moment (36% said yes, 17% said don’t know and 1% refused to answer).

The findings of the IDFI suggest that Georgia has quite a way to go to achieve governmental openness. According to IDFI, a more open and interactive approach to public information on governmental websites may increase institutional trust, promote active citizenry as well as a belief in democratic government. In addition, a stronger desire for such information from the population and an interest in government activities may encourage more open government. What do you think?