Happy holidays 2018

This year, CRRC-Georgia’s carried out a wide variety of research on Georgia as well as the South Caucasus. If you’re looking for a social science read or a new dataset to play with over the holidays, here’s a few from CRRC-Georgia from 2018.

1. Knowledge and Attitudes of the Population of Georgia towards the Judiciary

What does the public think of the judiciary? Have the reforms of the Georgian Dream government led to increased trust in the courts?  This report looks at the knowledge and attitudes of the population towards the judiciary and changes between 2014 and 2018. To read the full report (in Georgian), click here, and to read a briefer version in English, click here. To explore the data, click here.

2. Countering Violent Extremism in Georgia: A Needs Assessment

This year, CRRC-Georgia carried out a needs assessment of communities that have lost members to the conflicts in Syria and Iraq. The needs assessment asked two primary questions. 1) What distinguishes communities that have lost members to the conflict and have not that are otherwise similar? 2) What do people in communities that have lost members to the conflict want for their communities? To answer these questions, CRRC-Georgia used a matching algorithm to identify the most similar communities in the country to those that have lost members and carried out a survey in both types of communities. To see what we found, click here for the report and dataset.

3. Anti-western Propaganda Barometer

What’s the far right talking about in Georgia? LGBT people. Well, not only that, but really, they are talking a lot about LGBT people, according to the new dashboard CRRC-Georgia built in 2018 which provides weekly updates and analysis of what far right groups are talking about on Georgian language Facebook using a variety of machine learning tools. To explore the data, click here.

4. Elections, elections, elections

On Sunday December 16, Georgia’s first directly elected female president Salome Zourabouchvili was sworn into office after winning in a second round run-off. During the first round of elections, Senior Policy Analyst David Sichinava and Researcher Rati Shubladze built pollster.ge, which provided analysis similar to Nate Silver’s 538 in the lead up to the elections. One aspect of the campaign that was widely commented on was its negativity. For the second round, in an article for OC-Media, Koba Turmanidze explained the phenomenon by pointing out that it likely worked, at least for the opposition. On election night, Deputy Research Director Dustin Gilbreath gave his hot take on the elections to Euronews, and David Sichinava summed up the elections the following morning also on Euronews.

5. Citizen Participation in Policy Formulation at the Local Level

Have a decentralization wonk in your life? Well, this is the report for them. As Georgia continues towards devolving powers to local administrations, CRRC-Georgia prepared this report on citizens’ participation in local government in Georgia to help inform local government efforts to transition into a more empowered role in citizens’ lives. To read the report (in Georgian), click here.

From CRRC-Georgia, we wish you a happy holiday season, and Social Science in the South Caucasus will be back in January. To stay up to date with what’s happening at CRRC-Georgia, follow us on Facebook or sign up for our monthly newsletter Data Bites.

2017 Caucasus Barometer Data Release

This week, 2017 Caucasus Barometer survey (CB) data will become publicly available on CRRC's online data analysis portal. CB is the longest running survey project in the South Caucasus region, with data available from 2008 to present. It enables the comparison of trends in the region over time. Caucasus Barometer 2017 was carried out in Armenia and Georgia in Fall 2017. To view the data for both countries or download the data sets, check our online data analysis platform from February 1.

Senior Policy Analyst David Sichinava discussing CB 2017 results at CRRC-Georgia's office.

Get credible! ... Or a modest proposal to implement pre-registration in think tank research

[Note: This post first appeared at On Think Tanks. It was written by Aaron Erlich Assistant Professor of Political Science and Founding Member of the Centre for Social and Cultural Data Science at McGill University and Dustin Gilbreath a Policy Analyst at CRRC-Georgia. The Caucasus Research Resource Centers in collaboration with Aaron Erlich and Caucasus Survey recently announced a pre-registration competition for articles that will use the 2017 Caucasus Barometer Survey. This post is a reflection on how and why think tanks can and should use pre-registration based on the experience of setting up the competition and a summer workshop on pre-registration hosted at CRRC-Georgia in summer, 2017.]

It’s almost a cliche to say that think tanks operate on the basis of credibility. The media, politicians, and some in the general public have increasingly questioned think tanks’ credibility in recent years, with think tanks and tankers becoming increasingly thought of as lobbyists under a different name. Think tanks are not the only ones experiencing a credibility problem. Social science outside the think tank world is also in the middle of a credibility crisis. This crisis stems from the lack of reproducibility of results, scandals related to data fabrication, reliance on small sample studies, and questionable data analysis practices in the search of statistical significance. In response to this crisis, one proposal that aims to ameliorate the situation is the pre-registration of studies. Pre-registration not only represents an opportunity for social science, but also for think tanks to increase the credibility of their work, lighten workloads, and increase donor independence.

What is pre-registration?
A pre-registered study is one where research design elements like sample size, hypotheses, any experimental protocols, and statistical analyses are defined, justified, and placed in a secure registry prior to actually carrying out data analysis. Usually, this means registering the study prior to data collection. However, in some cases one can pre-register a design while data collection is ongoing or before the data are available to the researcher.

For example, the Caucasus Research Resource Centers (Aaron’s former employer and Dustin’s current) is currently holding a competition for papers on foreign policy preferences in the South Caucasus based on the 2017 Caucasus Barometer survey. The survey, at the time of writing, is entering the field and is expected to be complete at the end of October. However, the data itself will not be released until December. To participate in the competition, researchers must register their research design at the Open Science Foundation registry (one of a number of reputable locations to register a study) and then submit their paper based off their pre-registration (without results) to the journal Caucasus Survey. The papers will be reviewed and accepted or rejected without the results of analyses, hence taking away the incentive to find statistical significance.

Like in other studies, analysis commences once an organization has collected data. However, in contrast to an unregistered study, after data collection a researcher need only focus on carrying out the analysis described in their pre-registration plan (or even simply run pre-written code for analysis) and insert the tables and graphs into their report. The bulk of the registration document serves as the report, hence front-loading the writing. Any exploratory data analysis, not described in the pre-registration, is reported as such in the final report.

Why would a think tank pre-register a study?
From the perspective of a think tank, there are reputational advantages as well as more subtle bonuses for managing a think tanks’ workload. The most important advantage of pre-registration will likely be that it increases the credibility of the think tanks’ findings. While in the past, research consumers often simply considered quantitative work robust, today issues surrounding replicability, and statistical modeling like hacking data for statistical significance at the holy 5% level have cast a large shadow over a great deal of quantitative work. Pre-registration precludes statistical hacking among other issues like researcher degrees of freedom, thus leaving fewer avenues of attack for potential critics.

Besides increasing credibility, pre-registration of research design can be particularly valuable for think tanks who work on commissioned studies. While philanthropy is one of the main sources of funding in the United States, in the developing world, think tanks often survive on service contracts and grants for studies on specific issues. Pre-registration is beneficial for both the academic reasons outlined above as well as for the think tank’s work load and independence.

When it comes to workload, a pre-registered study shifts a great deal of effort to project start up, but has the potential to decrease the ultimate workload. Because hypotheses are specified ahead of time, researchers will have to start writing out their expectations instead of focusing solely on design at the start of projects. This means that clients need to agree beforehand on what analyses the think tank will perform, and that agreement can be put into the deliverables of any contract. In this manner, donors who contract research could be constrained in their ability to request more analyses at the end of the project (at least without paying and specifying that these were not pre-registered results). In turn, this prevents researchers from needing to run (potentially hundreds) of additional analyses at the end of a project, when the client is unsatisfied with the results for whatever reason or curious about some other result they had not thought of ahead of time.

When it comes to independence, in the current environment, many donors do not hesitate to pressure researchers to produce results supportive of donors’ positions. With a pre-registered study, researchers have listed out the exact analysis they will implement beforehand along with their expectations about the results of the analysis. Donors who have have been educated in the way this process works can use pre-registration to make stronger arguments. Moreover, the analysis can be built into the deliverables of the project. Hence, donors will be less facilely able to suggest a different analysis or measure in place of the one the researcher chose at the start, thus decreasing the number of avenues through which donors can apply pressure, particularly since changing the analysis would have cost implications.

Many in the think tank community might be skeptical of this proposal. Afterall, donors hold the purse strings. Still, we think that the process will benefit many donors. In many cases, donors want the highest quality study possible, even when motivated by short term goals, because it will help them advance their agenda. While if the donor is so baldly seeking a single result and not willing to accept anything else, we accept the fact that the proposal won’t work. In many case, however, we think donors would take enhanced quality at no cost in exchange for a loss of some control over the final product.

Pre-registration does have limitations and drawbacks. For example, it only works when a researcher can credibly demonstrate that they do not have access to data before pre-registering. Moreover, although it is likely the model could be applied to qualitative research in some form, to date, the model has yet to be implemented widely. Since qualitative research arguably comprises the majority of think tank research, the scope of use is somewhat limited.

While pre-registration is not panacea to the problems of social science or the problems think tanks face, it is a tool for think tanks to consider, which can enhance credibility and potentially decrease workloads and increase independence - three things we don’t think many tankers would be against.

Dustin Gilbreath is a Policy Analyst at CRRC-Georgia and the Communications Manager at Transparify. 

Aaron Erlich is an Assistant Professor of Political Science and Founding Member of the Center for Social and Computations Data Science at McGill University, and previously a Research Consultant at the Caucasus Research Resource Centers in the Tbilisi office.

CRRC’s Fifth Annual Methodological Conference: In Search of Methodological Innovation

CRRC’s fifth annual Methodological Conference took place on June 23 and 24, 2017 in Tbilisi. This year the conference’s focus was on policy analysis in the South Caucasus, and the search for methodological innovation. Over 50 participants representing institutions in the United States, United Kingdom, Georgia, Azerbaijan, Armenia, Russia, and Canada attended.

Alexis Diamond of the Keck Graduate Institute (KGI), San Francisco gave an opening address for the conference titled: Deliberate ignorance: The dangers of knowing too much too soon. The talk covered a wide range of issues in evaluation, however, emphasis was placed on the importance of honest evaluation.

The first day of the conference had four sessions, with papers on a wide variety of subjects from the geographies of polarization and inequality in Tbilisi to a field experiment on marshrutka safety and a machine learning approach to profiling tax awareness in Armenia. 

The opening slide of David Sichinava’s presentation on Spatial Patterns of Emerging Inequalities in Tbilisi, Georgia.

The second day of the conference was dedicated to workshops. Alexey Levinson of the Levada Center, Moscow lead a workshop on open-ended group discussions and Aaron Erlich of McGill University discussed the fundamentals of multiple imputation. The conference also included workshops on case studies in public health, web surveys, and synthetic controls.

Aaron Erlich discussing why and when to use multiple imputation.

For more information, the full conference program can be accessed here.

Perceptions of closeness: Opinions about which regions’ traditions and modern culture are closest to Armenian’s and Georgian’s

CRRC’s 2015 Caucasus Barometer (CB) survey asked the populations of Georgia and Armenia to identify the regions in the world whose traditions and modern culture are, in their opinion, closest to Armenia’s and Georgia’s. While most people in both countries feel closest to neighboring countries, many found it difficult to answer the questions. While there are many commonalities in responses to these questions in Armenia and Georgia, there are also minor differences.

Two questions were asked in CB. One focused on traditions and the other on modern culture. In both cases, the same show card was offered to the respondents. When it comes to traditions, the populations of both countries named most often “neighboring countries in the South Caucasus”, but this answer was more common in Armenia (36%) than in Georgia (27%). Interestingly, in both countries people found it similarly difficult to answer this question. Roughly one third of Armenians (30%) and Georgians (36%) could not or refused to answer the question.

Besides similarities, there are also differences. The “European part of Russia” was the second most common answer in Georgia (15%), and the third most common in Armenia (8%). In Armenia, “Neighboring countries in Asia” was the second most common answer, although the difference in frequencies of naming these regions in Armenia is within the margin of error. It is clear though that the population of Georgia generally does not consider the traditions of “Neighboring countries in Asia” to be close to Georgian traditions.  

The responses did not change much when people were asked which region’s modern culture they considered closest to their own. Slightly more Armenians (30%) named neighboring countries in the South Caucasus than did Georgians (23%), and the “European part of Russia” was, again, the second most common response in Georgia (13%) and third in Armenia (8%), with even higher shares of those answering “Don’t know”. Importantly, comparison of the answers to these two questions suggests that the populations of these countries do not see important differences between the traditions, on the one hand, and modern cultures, on the other hand, of the regions worldwide. 

Interestingly, in both countries, responses are quite similar in different settlement types (Georgia, Armenia), age groups (Georgia, Armenia), and by gender (Georgia, Armenia).

Despite minor differences, the populations of Armenia and Georgia have similar views about their cultures’ propinquity to other cultures in the world. Yet, in both countries a large share of the population find it difficult to answer the respective questions.

The datasets used in this blog post and related documentation are available at CRRC’s online data analysis platform. 

Trends in Scientific Output in the South Caucasus: 1996-2012

Increases in the quantity of peer reviewed publications, on the one hand, and citations, on the other hand, are believed to measure scientific progress. In the 1960s, efforts to document, explore and explain trends in scientific progress gave rise to the quantitative study of science and science policy – Scientometrics. The SCImago Journal & Country Rank Portal covers the period from 1996 to 2014 and provides indicators that help us examine the dynamics of scientific output worldwide, including in the countries of the South Caucasus (SC). In this blog post, we present the following indicators: 

• Number of publications in peer reviewed journals by academics affiliated with the country’s scientific institutions;
• The ratio of cited peer reviewed publications to uncited ones;
• International collaboration, measured by the ratio of peer reviewed publications co-authored by individuals from different countries.

The number of peer reviewed publications by authors from all three South Caucasus countries has increased between 1996 and 2012. The chart below shows a notable increase after 2002.

Importantly, the number of both cited and uncited publications has been increasing in each scientific discipline. The ratio of cited to uncited publications, however, has declined slightly, as the chart below shows. 

At the same time, international collaboration, measured as the ratio of SC authors’ peer reviewed publications co-authored with authors from other countries, seems to have increased between 1996 and 2012. As the chart below shows, Azerbaijan falls behind, while international collaboration is slowly but steadily increasing in Armenia and Georgia. 

To sum up, the number of peer reviewed publications is the most rapidly growing indicator, suggesting positive developments in the scientific community of the countries of the South Caucasus. However, it seems that there is still work to do when it comes to the impact of the publications by scholars in the South Caucasus.

To learn more on the subject, take a look at this comparative analysis of the scientific output of Armenia, Azerbaijan and Georgia; this study which examines trends in the development of science in the CIS countries, and this article (in Georgian) focused on Georgian academic institutions. 

Georgian society’s attitudes towards forced ‘volunteering’

Although the history and contextual background of volunteering in Europe vary from country to country, in most western societies, the development of volunteerism is associated with the emergence of civil society organizations, which appeared on the scene as a result of the crisis of the welfare state when governments could no longer manage to respond to the growing needs of society. Until the end of the 1990s the situation was different, though, in many socialist states, where the citizens were often forced to ‘volunteer’ for state controlled organizations. This practice had nothing to do with actual volunteering and would sometimes lead to human rights abuses. At the time, one of the common forms of this phenomena was ‘community service,’ e.g. organized cleaning of large public spaces, usually carried out on Saturdays (Shabatoba in Georgian). For many people living in socialist countries, it was an obligatory activity, often of a political character, and widely used by the state propaganda machine.

A number of recent studies argue that currently observed negative attitudes towards volunteering in some of the post-socialist societies are a consequence of Shabatoba and similar experiences (UNV, 2010; GHK, 2010; Ekiert and Foa, 2011). At the same time, there are other studies (UNV and UNDP, 2009) arguing that it is impossible to generalize any finding about how people felt and still feel about forced 'volunteering' practices.

Currently, attitudes towards Shabatoba-type activities are all but straightforward in Georgia. This blog post looks at how these attitudes differ by age, level of education, and previous volunteering experience, based on the findings of CRRC’s April 2014 Volunteering and Civic Participation in Georgia survey, funded by East-West Management Institute (EWMI) and the United States Agency for International Development (USAID).

According to the abovementioned UNV and UNDP 2009 study, if older generations tend to remember socialist era unpaid work programs with certain affection, for the younger generations, it is often hard to understand how such forced work could have been enjoyable. Considering this finding, one would expect younger Georgians to have less positive attitudes towards Shabatoba-type activities, compared to representatives of older generations. Surprisingly, however, the survey results show that representatives of different generations report quite similar attitudes towards potential forced ‘volunteering’ in today’s Georgia. The majority of Georgians of all age groups agree with the statement that “Georgia would be better off today, if the government forced us to volunteer,” although a slightly smaller share of young people (18-35 years old) agree with it, compared with those aged 36 and older.

Note: A 10-point scale was used to record answers to the question, the answer options for the question “In the Soviet Union there were Shabatoba-s, when people were jointly cleaning up public spaces. What do you think about having similar activities in Georgia now?” On the original scale, code ‘1’ corresponded to the opinion “We are lucky that the government does not force us to volunteer,” and code ‘10’ corresponded to the opinion “Georgia would be better off today if the government forced us to volunteer.” For this blog post, the answers were grouped as follows: codes 1 through 4 were labeled as “We are lucky that the government does not force us to volunteer.” Codes 5 and 6 were labeled as “Neither agree nor disagree with any of the statements.” Codes 7 through 10 were labeled as “Georgia would be better off today if the government forced us to volunteer.” Options “Don’t know” and “Refuse to answer” were excluded from the analysis. 

A number of studies conducted in the EU and in some of the post-socialist states (although not in Georgia) have shown that people with higher education tend to volunteer more than people with secondary education (UNV and UNDP, 2009; GHK, 2010). One might expect that the opinions about whether a government should be forcing citizens to volunteer or not would also differ by level of education. In fact, according to our survey results, those with higher education are slightly less inclined to agree with the statement that it would be better if the government forced citizens to volunteer, compared to those with secondary technical or secondary or lower education.

Note: The answer options for the question, “What is the highest level of education you have achieved to date?”  were grouped as follows: options “No primary education”, “Primary education (either complete or incomplete)”, “Incomplete secondary education”, and “Completed secondary education” were grouped into “Secondary or lower.” Options “Incomplete higher education”, “Completed higher education (BA, MA, or specialist degree)”, and “Post-graduate degree” were grouped into “Higher than secondary”.

Notably and rather unexpectedly, those who have previously been engaged in volunteering (for example, have planted a tree outside their own property or have cleaned a public space) report slightly more often that Georgia would be better off today if the government forced citizens to volunteer, compared with Georgians who have not participated in such activities.

Note: A new variable was computed for this chart based on the questions, “Have you planted a tree outside your property during the past 6 months?” and “Have you helped to clean public space during the past 6 months?”  The answer “Yes” in the new variable means participation in at least one of the abovementioned activities, and “No” means participation in neither. 

To sum up, the findings presented in this blog post are rather unexpected. The majority of Georgians think the country would be better off today if the government forced citizens to volunteer, and this attitude varies only slightly by age and level of education. Probably the most unexpected of these findings is that those with volunteering experience support this idea slightly more often than those who have not been involved in volunteering activities previously, and this correlation proves to be statistically significant, although relatively weak.

To learn more about volunteering in the South Caucasus, take a look at earlier blog posts, What We Know About Volunteering in Georgia, Volunteerism in the South Caucasus and Community support and volunteerism in the South Caucasus.  If you wish to know more about Volunteering and Civic Participation in Georgia surveys, visit our Online Data Analysis tool or take a look at the Volunteering and Civic Participation report.

Household income and consumption patterns in Georgia

After the collapse of the Georgian economy in the 1990s, the country slowly started to recover, and between 2000 and 2014, the gross national income grew from $3.4 billion to $16.7 billion (in current USD). According to the National Statistics Office of Georgia, the official unemployment rate in Georgia was 12.4% in 2014, but according to numerous surveys the rate is much higher. Compounding matters, the low salaries of the majority of those who are employed make it difficult for many families to make ends meet. According to World Bank statistics in 2012, 15% of Georgians still live below the national poverty line, which is slightly under $1.25 a day.

Classic micro-economic theory tells us that consumption depends on income levels, but also on the type of good. Normal goods, a good example of which in Georgia would be an Opel Astra if we are talking about cars, are consumed less when income falls. Inferior goods, on the other hand, are consumed when consumers do not have enough money to buy something better – with cars, one would choose a Zaporozhets instead of an Opel. Or, when a family’s budget is tight, they might buy no-name sports shoes rather than Nikes, or a standard mobile phone rather than a smart phone. Finally, demand for luxury goods (a Land Rover instead of an Opel) increases more than proportionately as incomes increase.

Recently, we saw that most people in Armenia, Azerbaijan, and Georgia cannot afford to buy certain durable goods. But, when income is scarce, it also obviously influences everyday choices, such as what to eat for dinner, when to turn on the heater, and whether to buy your child that chocolate bar she really wants.  

So how does the consumption of everyday goods differ by income level in Georgia? The chart below shows the percentage of households in Georgia which report restricting consumption of certain foods because of their limited income.

Note: The original scale measuring household income was recoded for this chart. GEL was converted into USD. The income groups “USD 401-800”, “USD 801-1200” and “More than USD 1200” were combined into “More than USD 400” and the groups “USD 0”, “Up to USD 50” and “USD 51-100” were combined into  “Up to USD 100”. Only “yes” answers are reported. Error bars indicate a 95% confidence interval.

As expected, the percentages of households limiting consumption differ by income group, with households in the highest income group limiting their consumption the least. Importantly, this trend is obvious in spite of the fact that we did not control for the size of households for this blog posts, i.e. did not take into account per capita income of the household members, or number of children or elderly in the households.

Even in the group with the highest household income, about one in four households limits consumption of beef – and reports doing so solely due to budgetary difficulties. In the $251-400 income group, between 8% (potatoes) and 54% (beef) of Georgians say they restrict consumption of certain goods. In households with an income between $101-250 these shares increase to 14% and 76%. In the poorest households, with less than $100 income per month, four out of five households report limiting consumption of beef and one out of five limits even the consumption of staple foods like potatoes.

In each income group, the highest percentage of households limits consumption of beef and sweets/chocolate, while the smallest percentage limits consumption of potatoes. Milk and vegetables are somewhere in between. In addition to the obvious explanation suggesting that higher shares of households limit consumption of beef and sweets/chocolate simply because these are relatively expensive, this trend might also be explained by what economist call income elasticity of demand. The concept is a measure of how sensitive consumption is to changes in income. Among food products, beef and sweets are probably seen more as luxury rather than absolute necessities, because of comparatively high prices and potential substitutes.

Hence, how Georgians limit their food consumption differs by both income group and type of food. Yet, looking at this issue only from the monetary income perspective does not take into account cases when, in rural settlements, families produce their food themselves.

Interested in this or a similar topic? Browse the Caucasus Barometer and other CRRC survey data here.

Common challenges, common solutions

[Editor's note: This is the sixth in a series of blog posts co-published with On Think Tanks. The views expressed within this blog series are the authors alone and do not represent the views of CRRC-Georgia.]

By Dustin Gilbreath

So far, in this series think tankers working in the South Caucasus have reflected on the issues challenging their countries’ think tank sector. In many ways, some fundamental problems lie at the heart of the specific problems, and I think they can more or less be summed up as problems with language and audience; quality of research; funding; and transparency. This post takes a look at one of these challenges – language and audience – and considers some things that might nudge the region’s think tanks forward.

Language and audience

Language, and specifically the demand for English outputs from donors, limits the size of the audience of research in the region. Zaur Shiriyev has described how the use of English in Azerbaijan in the 90s (and presumably to this day to a large extent) limited the public’s access to research, and Jenny Patruyan reflected on English-centric nature of think tank websites in Armenia. Definitely, different phenomenon, but language is still the underlying problem, and both authors’ issue with language comes from the fact that only an elite or foreign audience can access the research. Notably, funding was cited as one of the reasons for the English language outputs, and donors might help address this problem by requiring publications in both languages (and of course, should also fund translation and/or editing if they do).

When it comes to audience, I don’t think any of the contributors to this series have bemoaned think tanks’ efforts to reach elites so much as highlighted that organizations should consider broadening the reach of their research rather than targeting elites alone. To me at least, expanding to a broader audience seems like a good idea, maybe not for all organizations, but for many. To do this, first everything has to be in an accessible language, but just as importantly it should be in a form that someone will actually consume – only the most dedicated reader will take the time to go through a 50 page policy paper.

This doesn’t mean that we don’t need policy papers anymore, but rather that think tanks here should try to pair their longer, more demanding of the reader outputs with simpler and more accessible ones. Infographics and even products think tanks wouldn’t normally consider producing like games should be options that are on the table. Some progress has been made on the digestibility front, and Jumpstart Georgia’s work provides strong examples for other organizations in the region.

Something that would not only help with the language/audience problem, but also probably contribute to developing quality would be the development of something resembling Think Tank Review. Although the original was spurred on by the need to get policy makers to actually read reports, the organization, in practice, also spreads, archives, and reviews think tank work. For the South Caucasus, there would need to be translation into local languages (and potentially Russian) on top of TTR’s usual work, but language aside, it could improve quality by letting researchers know their work could be reviewed. As internet access is prevalent throughout the region, and most 18-55 year olds here know how to use the internet, something like TTR could bridge the elite to general public gap. Notably, a regional site would help this divided region stay better informed about the goings on of their neighbors, and could serve as a platform for discussing the larger issues facing the South Caucasus as a region rather than as individual countries. Moreover, policy success could be shared and reflected upon.

Of course, these are just a few ideas, which might make dents in the problems described so far in this series. Have other thoughts? Let’s have a conversation in the comments section below.

Do Think Tanks in Georgia Lobby for Foreign Powers?

[Editor's note: This is the fifth in a series of blog posts co-published with On Think Tanks. The views expressed within this blog series are the authors alone and do not represent the views of CRRC-Georgia.]

By Till Bruckner

If you work on policy issues in a transition or developing country, you probably know the standard line on think tanks by heart. Local think tanks build domestic research capacity, improve policy formulation processes and outcomes, and enrich and enhance democratic debates, thereby contributing to the emergence of more democratic, wealthy, and equitable societies. (Yes, you may copy and paste this into your next fundraising proposal if you wish).

Below, just for the heck of it, I’ll make the counterargument. Here’s the rival hypothesis:

Most local think tanks in transition or developing countries do not conduct real research. They just act as front groups for foreign lobbying campaigns. Western powers fund them to capture domestic policy formulation processes and distort democracy. Acting in concert with equally remote-controlled, faux-local NGOs and media, think tanks use foreign funds to push foreign agendas, creating a heavily tilted playing field on which the politics and policies favoured by the West always come out top, and on which real democracy can never emerge.

In September 2014, when the New York Times alleged that foreign donors were attempting to “buy influence” by bankrolling major think tanks in D.C., it caused an outcry across America. Congress rapidly reacted and soon after adopted a new rule requiring Congressional witnesses employed by think tanks to reveal any foreign funding that they and their parent organization had received. One lawmaker pushed foreign funding for non-profits onto the agenda of an upcoming review of foreign lobbying safeguards. A whole year after the event, some think tanks still seem to feel a bit shaken by the whole hullabaloo.

These revelations were enough to spark a protracted and sometimes paranoia-tinged debate on how foreign funding of think tanks might actually be foreign lobbying in disguise, and how such sinister practices threaten to distort American democracy and exert undue influence on foreign policy formulation in D.C. 

Ironically, America was at the same time conducting sustained, highly orchestrated and lavishly funded foreign lobbying campaigns via think tanks to pry the countries of Eurasia away from Russia and to build elite and popular support for a rival pro-Western path consisting of NATO membership, EU accession, and free market reforms. 

The thing is, nobody in Eastern Europe, the Caucasus or Central Asia called them lobbying campaigns. Instead, we called them “democracy promotion”. Or “civil society strengthening”. And American funding for local think tanks and local NGOs – the borders are blurred as both tend to tank far more than they think – was a key element of these lobbying attempts.

The following description of USAID’s More Transparent and Accountable Governance program in Georgia neatly illustrates how America’s foreign lobbying efforts work: pour money into think tanks, NGOs and the media – not to promote independent policy research and critical thinking, but to push pre-defined Western policy agendas:

“[T]he project will focus on strategic policy issues identified jointly by USAID and GoG partners… The project will provide support in advancing these issues through Georgia’s policy development and law-making systems and processes. In complementary fashion, support to CSOs and journalists under this project will align with the same set of strategic priorities where possible… Georgian CSOs, particularly think tanks, will receive training on similar skills under the civil society component of this project.”

While some pundits in America get paranoid about think tanks with a bit of foreign funding in their portfolios, USAID posts a comprehensive foreign lobbying strategy on its Georgian website – and nobody considers this unusual. 

Considering foreign actors’ role in Georgia’s Rose Revolution and Ukraine’s Orange Revolution, and USAID’s apparent willingness to use foreign aid as an inducement in explicitly quid pro quo foreign lobbying deals, this is surprising; even in bigger and more resilient Germany, some people are beginning to raise the alarm about possible democratic distortions as a result of American-funded non-profits’ maneuverings. 

Efforts by governments abroad to boost non-profit oversight and regulation tend to be dismissed almost reflexively as ‘crackdowns’ by many Western donors and their chorus line of NGO grantees, even in the case of mature democracies such as India. These opponents of meaningful government oversight frequently forget – or choose to ignore – that even the world’s oldest democracy, the United Kingdom, places restrictions on the political activities of non-profits, including those that are entirely domestically funded. 

If a foreign government designed and funded a similar programme in the U.S., the participating NGOs and think tanks would almost certainly need to register under the Foreign Agents Registration Act, forcing them to document and subsequently disclose their financial dealings and interactions with officials in great detail. 

Non-profit regulation is a minefield in every country, balancing the imperative to preserve freedom of speech (including donor-funded speech) on the one hand, and the desire to protect democratic debating space from excessive distortions by powerful and wealthy vested interests (specifically donor interests) on the other. 

(See here for an especially good attempt to strike such a balance through dialogue.)

In Georgia, two decades of largely unregulated, nearly 100% Western-funded think tanking have had positive outcomes in terms of policy, but arguably negative outcomes in terms of politics. 

On the positive side, think tanks have largely delivered on their promise in terms of policy development: they have helped to build domestic research capacity and to improve policy formulation processes and public debates around specific policies, and they deserve a lot of credit for this.

On the negative side, I have witnessed how, over time, think tanks in Georgia have narrowed the field for democratic debate. There’s something impoverished about a political landscape in which every single think tank paper seems to be pro-Western, pro-NATO, pro-E.U., pro-market, and even pro-gay rights, with the scope of inquiry reduced on to how best and fastest to join NATO, join the EU, open markets and privatize, and safeguard minority rights. 

Meanwhile, alternative voices, choices and options have been pushed to the margins of media coverage, public debate, and eventually electoral politics, in the process politically marginalizing Georgian voters with different ideological leanings by robbing them of voice, choice and representation. NATO-or-Putin, privatization-or-communism, gays-or-gulags – only idiots and traitors would dissent.

Some observers lament America’s polarized think tank landscape with its endless dialectic clashes between Heritage, Cato, the Center for American Progress & Co., but they miss that each side in every debate should get a chance to articulate, refine and communicate its views through rival think tanks. In Georgia, there is no such democratic competition, as only one ideological side has all the think tanks, all of which financially depend on a cabal of Western donors with overlapping agendas. 

What can Georgians do to reclaim and diversify their domestic political space? One option is the German model of funding for think tanks tied to political parties with public money. Another is for think tanks to mobilize more money from domestic donors. Either way, if Georgians want to have a healthy democracy, sooner or later they will have to start paying the bills. 

Till Bruckner spent several years living in Georgia, where he witnessed the Rose Revolution while shilling for sinister foreign interests in a variety of roles, including think tank ventriloquist, NGO stooge, and journalistic parrot. He later wrote his PhD thesis on accountability in international aid in Georgia, focusing on the political economy of aid and non-profit accountability. His collection of hats currently includes freelance journalism, think tanking, and running advocacy efforts for Transparify, an initiative promoting greater think tank transparency worldwide. This article was researched and written in a private capacity. None of the views expressed here should in any way be taken to represent the views of Transparify or of any other organization.

The views expressed in this blog post are those of the authors' alone and do not necessarily represent the views of CRRC.

The development of Azerbaijani think tanks and their role in public policy discourse

[Editor's note: This is the fourth in a series of blog posts co-published with On Think Tanks. The views expressed within this blog series are the authors alone, and do not represent the views of CRRC-Georgia.]

By Zaur Shiriyev

The development of local think tanks in Azerbaijan has taken a different route to that followed by most other post-Soviet states and Eastern European countries. In the Eastern Bloc countries, research institutes modeled on Western think tanks became increasingly popular following the collapse of the Soviet Union. However, in Azerbaijan this did not happen, largely due to domestic political developments in the early 1990s.

Barriers to development

During the latter years of the Soviet Union, Azerbaijani social science research institutions educated the public and raised awareness about the Nagorno-Karabakh conflict with neighboring Armenia as well as national history. They did so by publishing books and making media appearances, which had previously been subject to censorship and/or propaganda under the Soviet regime.

Members of academies and universities played a role in popular uprisings against the Soviet Union at the end of the 1980s, and held government positions immediately after independence. However, after a change in government in 1993, most of these people entered opposition politics rather than returning to academia or setting up research institutions that could promote new ideas or policy recommendations for the government.

Without a doubt, research capacity was also weak. Academics were unable to integrate Western social science methodologies into their discourses, and instead, old Marxist dogmas mixed with nationalist rhetoric prevailed. Therefore, despite their early role in public enlightenment, the research institutions did not develop.

Thus, the National Academy of Science – which encompassed over thirty scientific institutions and organizations – remained locked in Soviet tradition, whereby they served the state without any aim of playing a constructive role in stimulating or shaping public discourse.  This was the main difference between Azerbaijan and its neighbors, and from the outset weakened the role of research institutions in shaping the political agenda and public opinion.

These two factors – the lack of post-independence capacity building for old research institutions and a political environment that resisted the establishment of Western-style research organizations – were the main barriers to the development of the think tank sector during the first decade of independence. But a number of other factors also contributed.

In the early 1990s, Western funds were used to increase civil society capacity through the creation of NGOs, aimed at supporting democratic development. In pursuit of funding, NGOs called themselves “research institutions” or “think tanks” to increase their chances of securing funds. This led to the creation of increasing numbers of so-called think tanks in Azerbaijan. All that was required by foreign donors from these self-declared think tanks though was official NGO registration status. But, with a tiny number of employees and limited, grant-based funding, these NGOs struggled to fulfill their promises. The pursuit of funding also led to the “one man think tank” phenomenon, and in 2014, there were more than 3000 NGOs in Azerbaijan, and roughly 70-120 of them used ‘think tank’ or ‘research institution’ in their name.

This quasi-think tank community lacked Western-trained academics and experienced scholars, and the standard of work tended to be relatively low. Adding impetus to the problem, rapidly changing donor preferences in regard to subject matter meant that no one had a chance to develop real expertise in any one area (This also took place in Georgia, as observed by Ghia Nodia).

The factors outlined above posed significant institutional challenges, but perhaps even more important was the extent to which think tanks were prevented from participating in public discourse.
Until the end of the 1990s, media censorship limited the public appearances of researchers. Compounding matters, researchers were often working on topics that did not attract public interest. Moreover, funding often came with the stipulation that reports be produced in English, which seriously limited organizations’ audiences to a more educated and informed public.

When experts did appear in the media to discuss political issues, they tended to display partisanship, rather than presenting objective analysis. Weak analytical research skills and the absence of any real policy dialogue with the government undermined this community in the eyes of the public. Clearly, think tanks needed to cooperate and engage with political elites, but the poor relationship between the two sectors prevented this from happening. Moreover, the government did not seek the advice of research institutions, as they lacked the capacity to provide input on, for instance, the development of the economy and energy sectors. Instead, it turned to international expertise. Thus it was international experience that supported the creation of key national institutions like the State Oil Fund of Azerbaijan. The government also needed to increase its influence in Western capitals, especially in the US. While policy research institutes could have played a valuable role here, the government, seeking what it saw as an easier and cheaper alternative, invested in lobbyists.

Progress

The development of the expert community’s capacity began in the mid-2000s and stemmed from government initiative and greater human capital. On the governmental side, this process was initiated due to its greater financial resources, its need for institutions to advise it, and the desire to build and integrate a pro-government research community into its overseas lobbying strategy.
In 2007, the establishment of a government funded think tank (the Center for Strategic Studies, or SAM) marked the start of this process. In the same year, Parliament approved the concept of state support to NGOs, establishing the Council on State Support to Non-Governmental Organizations under the President of the Republic of Azerbaijan. The first NGO Support Fund, which gave small research grants to NGOs, required that grant recipients disseminate their research via media appearances. The establishment of the Science Development Foundation in 2009 increased funding opportunities for academic research. In prioritizing research activities, the government sought to improve its national and international image.

The government’s motivation’s aside, there emerged a group of people who had gained experience working at think tanks abroad, primarily in Turkey. Many of them had been educated in prestigious US and European universities. This new human capital dramatically changed things, and the new generation opened up a way to avoid working for government-funded think tanks and research institutions by establishing independent organizations.  But the early – unrealized – expectation was that this investment in human capital could lead to the establishment of a think-tank that could offer a model for others in future, as seen in Turkey. For instance, in Turkey, the 2000s saw the establishment of non-state think tanks- a key model in this regard was the Center for Eurasian Strategic Studies, or ASAM (Avrasya Stratejik Araştırmalar Merkezi); after that other think tanks proliferated based on that example.

In turn, this prompted foreign donors to take a more selective approach to grant allocation. They began to select institutions with the resources and capabilities needed to produce quality research. In 2008, Open Society Institute launched the “Program for Assistance to Analytical Centers”, aimed at improving the quality of political research and ensuring the sustainable development of independent political research institutions. Additionally, increased research funds from the EU within the European Neighborhood Policy framework and requirements for sector-specific research – e.g. on conflict resolution – created a more competitive environment for research institutions, leading to higher quality outputs.

All of the above enabled Azerbaijani think tanks to begin playing a larger role, which is demonstrated by the University of Pennsylvania’s Go-Think tank Report – in 2014, there were fourteen Azerbaijani think tanks listed, up from twelve in 2010.

Challenges

Notwithstanding the positive developments, there is still work to be done.
First, state funded think tanks and research centers have a strong tendency to orient their work towards international audiences, limiting local impact and capacity building, for instance, by improving access to policy research and debate by establishing high-quality policy publications in the Azerbaijani language.

Second, the parameters of enriching public discourse have shifted over the past decade. If in the 1990s, local media appearances were a priority for members of the expert community in stimulating public discourse, today, the combination of a proliferation of poor quality online media outlets together with increased informal state censorship of media has meant that most experts now prefer to publish in international media.

Television remains the main source of news for the majority of the population outside the capital, but expert participation is low. Particularly troublesome is the fact that there are no high-quality analytical programs that could serve as a bridge between the think tank community and the public which would broaden the scope of discourse and debate.

Third, legislative amendments to the Law on NGOs have placed serious limits on foreign funding, and most foreign funded NGOs operating as think tanks – regardless of the quality of their work – are struggling to survive. In general, the successful development of think tanks in international practice has relied heavily on philanthropy, which is absent in Azerbaijan. Thus in the absence of a competitive environment, which is further compounded by the lack of the financial resources, there has been a monopolization of the research market by a few experienced groups.

Conclusion 

The development of think tanks in Azerbaijan has faced serious challenges since the 1990s, when Western funding for the development of research centers lacked clear criteria and led the creation of quasi research institutions and one-man think tanks. The distinction between an NGO and think tank remains blurred. In general, think tanks played a minimal role in the shaping of public discussion during the 1990s.

The 2000s saw the creation of some higher capacity think tanks, but as before, the scope of the research was geared towards international audiences rather than the local public. As a consequence, high-quality policy journals and media outlets operating in Azerbaijani never developed. The shift to digital media led to the deterioration of research quality, pushing many professionals to publish in international journals rather than local ones, with the ultimate consequence of limiting the role of think tanks in public discourse.

Overall, the experience of Azerbaijan differs from other post-Soviet countries, especially in the former Eastern Bloc countries, not only because the latter benefited from greater Western investment in institution building, but also because in most of these countries the integration of scientific research into decision-making supported a smooth political process.

Currently, after more than two decades of experience in this field, there are a few main concerns and challenges for the development of the think-tank community that can be identified. The first is financial; without independent financial support, how can non-state think-tanks be developed? Second, given the proliferation of quasi think-tanks, what model for development can be followed, and specifically, can the establishment of research institutions in universities offer a better model for future development? Last but not least, taking into account that the integration of the non-state think-tank community into the policy making process remains impossible, what kind of steps should be taken to improve long term public outreach? One thought is establishing a common online platform for high-quality Azerbaijani language analysis. Have others? Tweet at me here.

Think Tanks in Armenia: Who Needs their Thinking?

[Editor's note: This is the third in a series of blog posts co-published with On Think Tanks. The views expressed within this blog series are the authors alone, and do not represent the views of CRRC-Georgia.]

By Yevgenya Jenny Paturyan

Think tanks are considered to be an important part of civil society: providers and keepers of expertise on important social, economic, environmental, political and other issues. Organizations like Chatham House and Carnegie Endowment for International Peace come to mind. In addition to ‘pure’ think tanks, there is a plethora of organizations that combine research with advocacy and action, Transparency International being a prominent example.

What does the think tank landscape look like in Armenia?

First of all, the term itself remains an alien catchword that has not taken root (and, frankly, perhaps it should not). Translated into Armenian (ուղեղաին կենտրոն) it sounds utterly ridiculous. Of the 11 Armenian think tanks mentioned by name in the Global Go Think Tanks Index Report 2014, in fact, only one describes itself as a think tank (in English, but not in Armenian). Some organizations prefer to call themselves research centers: a name that is easily translatable into Armenian. Others have broader descriptions corresponding to their general mandate (“an academic bridge between  diaspora [sic] and native Armenian scholars” or a “non-governmental organization … which aims to assist and promote the establishment of a free and democratic society in Armenia”).

Taking a closer look at those 11 organizations (supposedly the top Armenian think tanks as ranked by external reviewers) is an interesting reality check. As the 21st century saying goes: if you are not online, you don’t exist. How do these 11 names fare in a Google search? The table below summarizes the results.

Table 1. Online presence of Armenian entities mentioned by name in the Global Go Think Tanks Index Report 2014

*I did not search for a separate Armenian language web page. Instead, I simply noted if the main English web page has a link to an Armenian version

As a result of my little Googling exercise, one organization (Education and Training Unit) could not be found at all and another two are only mentioned on other organizations’ web sites but have no web sites of their own. Either they were active once upon a time, resulting in their names being listed in some databases, or they are very internet-unfriendly think tanks that prefer not to maintain their own web sites. (Let me clarify that at this point it is not my intention to engage in a discussion on how accurate the Global Go Think Tanks Rankings are, although this small experiment could be a starting point for such a debate. This was simply an empirical approach at trying to gauge the internet visibility of “top” Armenian think tanks).  Of the remaining eight organizations, two have not updated their websites since 2013. We are left with only six “finalists” that seem to be up and running. For those studying Armenian civil society this is no surprise. Many organizations in this sector are short lived or exist only on paper.

Who are the main consumers of those research centers’ outputs? Note that two of the six “finalists” (including the research center I work for) have no Armenian websites. This does not mean we do not produce Armenian language reports or policy briefs (we do), but it does tell you something about the main focus. Another interesting little experiment: of these four organizations that maintain both English and Armenian websites, if you just type in the ‘main’ website (like www.acgrc.am for our first example) it will take you to the English version in three cases; only in one case (http://ichd.org/) the first page you land on is the Armenian page. English language seems to be more important. This is no big surprise if you think where the main sources of funding are, but it does raise a question of how relevant think tanks are (and want to be) to the population of their own country.

One might argue that this is not a problem. Think tanks’ main “clients” are decision-makers. In the case of Armenia it should be the Armenian government and the international development organizations. Both turn to think tanks from time to time, but the outputs are produced for internal consumption, making it hard for the think tanks to establish themselves in the public eye, and to improve the quality of their products, as there is no equivalent of peer-review. As a result, Armenian think tanks remain virtually unknown to the public, including such important segments of the public as journalists, students, scholars, and others who would clearly benefit from think tank generated, systematized and stored information.

But is there a public interest in research and analysis produced by think tanks? Here is another little experiment: Civilnet (currently one of the leading sources of online news in Armenia) has about 30 articles in Armenian discussing research conducted by Armenian organizations or individuals: a simple search for the Armenian word հետազոտություն (research) returned 156 hits, out of which approximately every 5th was about research conducted locally. So, yes, there is some interest, and there are news outlets willing to publish think tanks’ stories. Of course researchers have to make an effort to translate their outputs into media- and public-friendly language (and of course, it wouldn’t hurt if it was also translated into Armenian).

While some Armenian think tanks are well established, there are many organizations (claiming to be think tanks) that are short-lived or active only from time to time. Their activity is mostly driven by external funding. They tailor their outputs more towards English readers. As a result, their public outreach and impact remains very low.

This short overview of the Armenian think tank landscape and their visibility online, leads to a number of questions:

1. Does Armenian public need to know more about the think tanks? Should think tanks prioritize public outreach more, or should they use their scarce resources to target donors and top decision-makers?
2. Do donors have a responsibility to share think tank outputs with the public in a language accessible to the public?
3. How can we ensure the quality of  Armenian language outputs, given that the circle of potential peer-reviewers is so small?

While these questions are beyond the scope of this post and are not likely to have simple answers, they are worth deliberating. What’s your take on these issues? Share your insights by tweeting at us here.

The lay of the land: An interview with Hans Gutbrod on think tanks in the South Caucasus

[Editor's note: This is the second in a series of blog posts co-published with On Think Tanks. The views expressed within this blog series are the authors alone, and do not represent the views of CRRC-Georgia.]

Interview by Dustin Gilbreath

Dustin Gilbreath: You recently recently pointed out that think tanks in the South Caucasus have come a long way in recent years, but that they still face challenges on some of the fundamentals – quality of research, policy relevance, funding, and operational acumen.  At the national rather than regional level, what are the relative strengths of and challenges before the think tank sector of each country?

Hans Gutbrod: The think tank sector is most relevant in Georgia, since Georgia is a context in which ideas are being worked out, and where there is an interest in policy solutions. Citizens have come to expect that the government delivers. So that, in principle, is a great opportunity for institutions in Georgia.

In Armenia, by contrast, policymaking is fairly closed. There are a few elite pockets of discussion, often involving the Central Bank and some other institutions, but in my view, the space for discussion is narrower. There are some bright spots, such as the Civilitas Foundation and CRRC Armenia. Both of those (and I have worked with both, to be clear) can contribute ideas, but the political system offers fewer access points.

Civilitas Foundation, interestingly, has developed into providing even more content and media, and that is a sensible approach, as it can be hard to transmit messages through traditional media that often has an insufficient institutional and financial basis for quality journalism. CRRC Armenia has been active on issues involving social protection for a long time, and has a good network and contacts in that field. Yet, overall, Armenia is punching way below its weight as a country, with only a limited effort to make it an attractive country to live, work and stay in. This limits the role of any research organization. Perhaps indicative of that situation is that even a Prime Minister whose appointment was greeted with at least some optimism, such as that of ex-Central Bank Director Tigran Sargsyan on balance delivered fairly little. When Prime Ministers can't deliver, there's not that much that think tanks can do.

In Azerbaijan, there isn't any think tank scene to speak of. The government typically has tried to solve any problem by throwing money at it, and by throwing any independent voice into jail. The results are mixed, at best. Yet, ironically, an investment in think tanks might be even more important, just for that reason. Ultimately it's possible, but unlikely, that the regime of Ilham Aliyev will last long. The most remarkable aspect about Ilham Aliyev really is how utterly incompetent his government is. They were handed huge amounts of money, and mostly blew it on themselves and a few prestige objects, instead of actually modernizing the country, its universities, and establishing alternatives to oil and gas. While the regime looks solid now, it has so little management capacity that it could unravel quite quickly in a crisis.

The key question is what alternatives will then be available. Given that there is no opposition to speak of, who can be ready to run things? You need to train people who understand the policy issues so that they can deliver results within the first six months of taking over. If you do that, a post-Aliyev government has a good chance. Conversely, if there is no viable alternative things could turn grim quickly, as Libya illustrates. I know this sounds far out right now, but it's important to hedge against downside risks, and thus policy research would be a good investment.

Coming back to Georgia, policy research organizations haven't really caught up with the new realities. The previous Saakashvili government was brimming with ideas. Some of these ideas were harebrained, but others also proved remarkably successful and even visionary. With the government hatching so many ideas, research organizations often struggled to keep up and had limited opportunities to contribute new suggestions.

This has now changed. Most people agree that the current government is much more receptive to outside input and ideas, partially because they produce fewer of their own. Yet we don't see that much input from research outfits. Many policy research organizations, have settled into a comfortable routine of criticizing the government, along with the society. That's understandable, but there is a missed opportunity of bringing in new ideas. Take one example: when the mayor of Tbilisi promised to plant 1 million trees, this would have offered an extraordinary opportunity. Research organizations could have jumped at this issue, coming up with ideas on how to plant these trees, talking about urban planning, reviving parks, greening the city, bringing in excellent ideas that worked elsewhere. Here and there this may have happened, but I haven't really seen a sophisticated paper by anyone that advanced the discussion.

So what's the biggest missing ingredient, for policy research organizations to succeed? In my view, it's curiosity. I myself have been to many events which are interesting, engaging, and where new ideas are being discussed. It's regrettable that often not a single policy researcher is there, even if the event happens close to their office. Improving ideas doesn't happen in isolation. It's a result of intense discussion. It also often requires sifting through lots of less relevant material. Of course, I understand that all these organizations have many other things to do: other projects, administration, other obligations.

Yet research outfits aspiring to be think tanks do need to ask themselves whether at the end of the day they really are passionate about policy. If you have not read Nudge, what are you doing at the table of a policy discussion? I know this is an extreme view, but I do think it needs to be brought into the discussion.

To be a good lawyer, to be a good practitioner, to be a good policy professional – for all of that you need to keep up with what's happening in the field, you need to connect with the community you work in. I wouldn't want my heart to be operated on by someone who is fumbling along, without having checked in with what's happened in the field in the last 20 years. Why should we demand less from think tanks? In the most extreme case, they're involved in open-heart surgery of entire societies.

Donors, too, should be discriminating in that regard, and hold local research organizations to a much higher bar. Asking for more transparency is just one of those aspects -not sufficient, but certainly necessary. We all need to ask more, to get better results, better policy proposals, and ultimately, better policies.

Ultimately I'm fairly optimistic that this can succeed, though think tanks may be dragged into the future rather than leading into it, and existing institutions may be sidelined. The playing field is now better for those that are agile. Ray Struyk has put forward a great book on how to manage think tanks, and this can help ambitious institutions get it right, if they take the materials seriously.

Another reason, to end on a plug for something that I put together, is that now journalists and ordinary citizens no longer need to take anything on faith. They have access to some of the best think tank research through tools such as www.findpolicy.org. So new ideas may come in, though not necessarily from formalized organizations.

The key challenge for all of us is how to accelerate a better understanding of policy. The best think tanks should be ahead of this process, not behind.

DG: In Azerbaijan, given the recent crackdown and shuttering of organizations which could provide just the training you mentioned, where are there (if there are) opportunities to invest in think tanks either from the side of donors or domestically?

HG: I think in Azerbaijan, the key is to invest in organizations that may work on the outside, that help to clarify what really is going on, that use innovative tools, that collect data that highlights how official data just isn't right. In a way this would be a research outfit that could feed into discussions via social media, a kind of research version of the original version of Radio Free Europe. I know that this is difficult, but you really need to invest into thinking precisely when times are particularly difficult. The case for such external research organizations has also been made by Emin Milli, a dissident who spent significant time in jail for daring to speak up.

DG: You noted here as you have noted elsewhere that donors need to push for higher quality research outputs from local organizations. Do you have any concrete recommendations of how to do so?

Well, there are many measures, and not so many have been tried. I would hope that donors engage substantially with this question. Here are some key measures that come to mind.

1. Finance: there should be more core financing to start with. Many research organizations are totally projectized, and this makes it harder for them to invest into quality.
2. Nudging: at the point of application, ask about quality assurance mechanisms. Some key questions could include what are you actually doing, with whom, how? Can you put this on to your website, to indicate your commitment to such quality assurance? That would be a good start.
3. Checklists: use and encourage the use of checklists. For example, does a policy proposal include a budget? Or is it just a wishlist? 
4. Comparison: encourage and finance small public reviews and comparisons. For example, does an organization make its old reports accessible, or does it lose all materials when updating a website? Unfortunately the latter still happens way too often, and is a great loss. Many organizations barely think of that, and knowing that you will be reviewed could encourage more attention on that issue.
5. Network: create access to constructive external peer review. You could potentially encourage the formation of a network of quality review. This would be an experiment, but if it works it could have a transformative impact. 
6. Transparency: there are many reasons for transparency, but an additional one is that having full financial information on a grantee website helps donor coordination. Donors should not just be transparent themselves, their quality can also be measured by the transparency of their grantees.  

I think there might be many more ideas, and that is why we need a debate on these issues.

DG: Any final thoughts?

HG: Yes. Ultimately, the really interesting things often are outside the disciplinary mainstream. I cannot recommend Ed Catmull’s book Creativity, Inc. highly enough. He describes how Pixar works, and how they consistently managed to turn out successful films. This book holds many lessons for creative organizations, and think tanks should be at least somewhat creative, and in the end also tell a compelling story.