Developing the “culture of polling” in Georgia (Part 2): The misinterpretation and misuse of survey data

[Note: This is the second part of a guest blog post from Natia Mestvirishvili, a Researcher at International Centre for Migration Policy Development (ICMPD) and a former Senior Researcher at CRRC-Georgia. The first part of this blog post is available here. This post was co-published with the Clarion]

The misinterpretation of survey findings is a rather widespread problem in Georgia. Unfortunately, it often leads to the misuse of data, which not only diminishes the importance of survey research, but also leads to more serious consequences for the country.

To illustrate how one might misinterpret survey data, the following example from CRRC’s 2015 Caucasus Barometer survey can be used. When asked, “What do you think is the most important issue facing Georgia at the moment?”, only 3% of the population mentioned low pensions, 2% the unaffordability of healthcare, and 2% the low quality of education. A number of issues including the violation of human rights, unfairness of courts, corruption, unfairness of elections, unaffordability of professional or higher education, the violation of property rights, gender inequality, religious intolerance and emigration were grouped into the category “Other”, because, in total, only 7% of the population mentioned these issues.

Based on these findings, one might think that these issues are unimportant in Georgia. However, this would be a misinterpretation, which happens for a number of reasons. Here, I focus on two. The first is:

1. Not paying attention to the exact formulation (wording) of the survey question, answer options, and instructions 

One reason a large share of the population did not mention the violation of human rights, gender inequality and religious intolerance as important issues is because each respondent could name only one issue. The options they chose (unemployment and poverty were named most often) were more important to them than human rights, gender inequality, and religious intolerance.

If a different question – “How important is the issue of human rights [or gender inequality, or religious intolerance] for Georgia?” – had been asked, the share of people who would answer that these issues are important would very likely be much higher than one or two percent. This wording would make people judge the issue not in relative, but absolute terms.

While working with survey findings, the exact wording of question(s) should always be taken into account. When the question is interpreted or reworded, it will almost inevitably lead to some degree of misinterpretation. More often than not, fieldwork instructions should also be taken into account. For example, was a show card used for the question? Was the number of answer options a respondent could choose limited or not?

Thus, it is crucial that survey results are understood and reported, keeping in mind the exact wording of the question(s), answer options provided, and any instruction(s) that had to be followed during the interviews. This will help minimize the risk of misinterpretation.

A second common cause of misinterpretation of public opinion polls in Georgia is:

2. Interpreting public opinion survey results as ‘reality’ rather than perceptions 

Even if the question discussed above had been asked so that the absolute rather than relative importance of the issues was measured and the survey findings still suggested that people thought the violation of human rights, gender inequality and religious intolerance were not important issues for the country, the findings should not be interpreted as a direct reflection of ‘reality.’ As discussed in the first part of this blog post, public perceptions are not ‘reality’.

Interpreting public perceptions as objective ‘reality’ is incorrect, because both perceptions and misperceptions, information and misinformation shape public opinion. It is equally important to remember that, sometimes, ‘reality’ simply does not exist. Moreover, as a number of studies have shown, it is often the case that people are simply wrong about a wide variety of things.

None of the above, however, diminishes the role and importance of public opinion polls. In fact, the misperceptions that survey findings can uncover are often among the most important outcomes for policymakers. Instead of putting an equal sign between public perceptions and ‘reality,’ data analysts and policymakers should critically analyze and address gaps between the two.

Going back to the above example, an accurate interpretation would consider the findings in the context of other studies that are specifically focused on human rights (or gender equality or religious tolerance). Indeed, numerous studies indicate that Georgia has serious problems with all three issues i.e., the population does not have much respect for human rights, gender equality, or people of other religions. Only looking at the latest Human Rights Watch report on Georgia makes this quite clear.

Looking at inconsistencies between people’s answers to different questions, or between survey findings and other types of data when available and relevant, is a good way to uncover misperceptions. For example, a 2014 CRRC/NDI survey found that roughly every fourth person reports there is gender equality in Georgia. However, about half of those who think so also think that taking care of the home and family makes women as satisfied as having a paid job, and that in order to preserve the family, the wife should endure a lot from her spouse.

The answers to these three questions should be presented and discussed not separately, as independent findings, but rather as interrelated findings that, taken together, give a better understanding of the assessments of and attitudes towards gender equality in Georgia. In this context, the question that needs to be raised and answered is why and how this inconsistency between answers occurs.

The misuse of survey findings happens when findings are presented and used in a way that reinforces people’s misperceptions and prejudices. The misinterpretation of findings often leads to their misuse, and eventually, can lead to serious issues.

Again, going back to the most important issue example, it would be a misuse of survey findings to conclude that since the violation of human rights, religious intolerance or gender inequality seem to not be perceived as important issues in Georgia, no policy is needed to address them. As demonstrated above, alternative sources show that these issues need to be addressed, and, at the very least, awareness of them needs to increase. Thus, policy intervention is needed.

What the survey findings tell us in this case is that people underestimate the importance of these issues. In turn, this contributes to the worsening of the problems. If you believe gender inequality or religious intolerance are not important, you probably would not care about these issues either. Thus, the larger is the gap between public perceptions and reality, the more important it is for policy makers to address the issue.

Public opinion should not be used as a directive for policy making without careful analysis of misperceptions and alternative sources of information.

Unfortunately, in Georgia sometimes it’s exactly the misperceptions that drive policy. Speaking of recent developments, misperceptions about homosexuality have lead politicians to talk more about the prohibition of same-sex marriage, something that has never been allowed in Georgia in the first place, than about human rights issues. Misperceptions about gender roles led politicians to reject a proposal that would define femicide as a premeditated murder of a woman based on her gender. Looking forward, the country cannot allow the misperception that the EU threatens Georgia’s traditions to drive the country’s foreign policy.

Now more than ever, when Georgia is still attempting to transition into a stable, democratic country, the country needs policymakers and researchers who have the knowledge and skills to critically analyze survey findings and use their potential for the development of the country.

Developing the “culture of polling” in Georgia (Part 1): Survey criticism in Georgia

[Note: This is a guest post from Natia Mestvirishvili, a Researcher at International Centre for Migration Policy Development (ICMPD) and former Senior Researcher at CRRC-Georgia. This post was co-published with the Clarion.]

Intense public debate usually accompanies the publication of survey findings in Georgia, especially when the findings are about politics. The discussions are often extremely critical or even call for the rejection of the results.

Normally criticism of surveys would focus on the shortcomings of the research process and help guide researchers towards better practices to make surveys a better tool to understand society. In Georgia most of the current criticism of surveys is, unfortunately, counterproductive and mainly driven by an unwillingness to accept the findings, because the critics do not like them. This blog post outlines some features of survey criticism in Georgia and highlights the need for constructive criticism aimed at the improvement of research practice, because constructive criticism is extremely important and useful for the development of the “culture of polling” in Georgia.

Often, discrepancies between the findings and the critics’ opinion about public opinion cause criticism of surveys in Georgia. Hence, the survey critics claim that the findings do not correspond to ‘reality’. Or rather, their reality.

But, are surveys meant to measure ‘reality’? For the most part, no. Rather, public opinion polls measure and report public opinion which is shaped not only by perceptions, but also by misperceptions i.e., the views and opinions that people have. There is no ‘right’ or ‘wrong’ opinion. It is equally important that these are opinions that people feel comfortable sharing during interviews –while talking to complete strangers. Consequently, and leaving aside deeply philosophical discussions about what reality is and whether it exists at all, public opinion surveys measure perceptions, not reality.

Among the many assumptions that may underlie criticism of surveys in Georgia, critics often suggest that:

1. They know best what people around them think;
2. What people around them think represents the opinions of the country’s entire population. 

However, both of these assumptions are wrong, because, in fact:

1. Although people in general believe that they know others well, they don’t. Extensive psychological research shows that there are common illusions which make us think we know and understand other people better than we actually do – even when it comes to our partners and close friends;
2. Not only does everyone have a limited choice of opinions and points of view in their immediate surroundings compared to the ‘entire’ society, but it has also been shown that people are attracted to similarity. As a result, primary social groups are composed of people who are alike. Thus, people tend to be exposed to the opinions of their peers, people who think alike. There are many points of view in other social groups that a person may never come across, not to mention understand or hold; 
3. Even if a person has contacts with a wide diversity of people, these will never be enough to be representative of the entire society. Even if it were, individuals lack the ability to judge how opinions are distributed within a society.

To make an analogy, assuming the opinions we hear around us can be generalized to the entire society is very similar to zooming in on a particularly large country, like Canada, on a map of a global freedom index, and assuming that since Canada is green, i.e. rated as “Free”, the same is true for the rest of the world. In fact, if we zoom out, we will be able to see that the whole world is all but green. Rather, it is very colorful, with most of the countries being of different colors than green, and “big” Canada is no indication of the state of the rest of the world.

Source: www.freedomhouse.org

People who think that what people around them think (or, to be even more precise – who think that what they think that people around them think) can be generalized to the whole country make a similar mistake.

Instead of objective and constructive criticism based on unbiased and informed opinions and professional knowledge, public opinion polls in Georgia are mostly discussed based on emotions and personal preferences. Professional expertise is almost entirely lacking in those discussions.
Politicians citing questions from the same survey in either a negative or positive context, depending on whether they like the results or not, is a good illustration of the above claim. For example, positive evaluations of a policy or development by the public is often proudly cited by political actors without doubting the quality of the survey. At the same time, low and/or decreasing public support for a particular party according to the findings of the same survey is “explained away” by the same actors as poor data quality. Subsequently, politicians may express their distrust in the research institution which has conducted the survey.

In Georgia and elsewhere, survey criticism should be focused on the process of research and should be aimed at its improvement rather than the rejection of the role and importance of polling. It is the duty of journalists, researchers and policymakers to foster healthy public debate on survey research. Instead of emotional messages aimed at demolishing trust in public opinion polls and pollsters in general, rationally and carefully discussing the research process and its limitations, research findings and their meaning/significance and, where possible, pointing to possible improvements of survey practice is needed.

Criticism focused on “unclear” or “incorrect” methodology should be further elaborated by professionally specifying the aspects that are unclear or problematic. Research organizations in Georgia will highly appreciate criticism that asks specific questions aimed at improving the survey process. For example, does the sample design allow for the generalization of the survey results to the entire population? How were misleading questions avoided? How have the interviewers been trained and monitored to minimize bias and maximize the quality of the interviews?

This blog post argued that survey criticism in Georgia is often based on inaccurate assumptions and conveys messages that are not helpful for research organizations from the point of view of improving their practice. These messages are also often dangerous as they encourage uninformed skepticism towards survey research in general. Rather than these unhelpful messages, I call on actors to engage in constructive criticism which will contribute to the improvement of the quality of surveys in Georgia, which in turn will allow people’s voices to be brought to policymakers and their decisions to be informed by objective data.

The second part of this blog post, to be published on January 23, continues the topic, focusing on examples of misinterpretation and misuse of survey data in Georgia.

CRRC’s Fourth Annual Methodological Conference: Research for Development in the South Caucasus

CRRC’s fourth annual Methodological Conference took place on June 24 and 25, 2016 in Tbilisi. Over 50 participants representing numerous institutions from seven countries attended.

David Lee, Chairman of CRRC’s Board of Trustees, opened the conference highlighting the importance of the issues discussed at the conference not only for the region, but also for the world.

The conference had a wide variety of workshops, such as Koen Geven’s  workshop on Causal Inference and Estimating Treatment Effects and  Julie A. George’s Methodological Approaches to Estimating Voter Fraud.

With four conference sessions focused on migration, politics, ideology and media, and gender inequalities in the labor market, the conference participants – academics and policymakers alike – had the opportunity to discuss the challenges with and ways forward towards generating more reliable knowledge on the issues.

CRRC-Georgia’s President, Koba Turmanidze, closed the conference noting that next year’s Methodological Conference will continue to focus on policy research and methodological issues, which can lead to better development policy in the region.

For more information, the full conference program can be accessed here.

Trends in Scientific Output in the South Caucasus: 1996-2012

Increases in the quantity of peer reviewed publications, on the one hand, and citations, on the other hand, are believed to measure scientific progress. In the 1960s, efforts to document, explore and explain trends in scientific progress gave rise to the quantitative study of science and science policy – Scientometrics. The SCImago Journal & Country Rank Portal covers the period from 1996 to 2014 and provides indicators that help us examine the dynamics of scientific output worldwide, including in the countries of the South Caucasus (SC). In this blog post, we present the following indicators: 

• Number of publications in peer reviewed journals by academics affiliated with the country’s scientific institutions;
• The ratio of cited peer reviewed publications to uncited ones;
• International collaboration, measured by the ratio of peer reviewed publications co-authored by individuals from different countries.

The number of peer reviewed publications by authors from all three South Caucasus countries has increased between 1996 and 2012. The chart below shows a notable increase after 2002.

Importantly, the number of both cited and uncited publications has been increasing in each scientific discipline. The ratio of cited to uncited publications, however, has declined slightly, as the chart below shows. 

At the same time, international collaboration, measured as the ratio of SC authors’ peer reviewed publications co-authored with authors from other countries, seems to have increased between 1996 and 2012. As the chart below shows, Azerbaijan falls behind, while international collaboration is slowly but steadily increasing in Armenia and Georgia. 

To sum up, the number of peer reviewed publications is the most rapidly growing indicator, suggesting positive developments in the scientific community of the countries of the South Caucasus. However, it seems that there is still work to do when it comes to the impact of the publications by scholars in the South Caucasus.

To learn more on the subject, take a look at this comparative analysis of the scientific output of Armenia, Azerbaijan and Georgia; this study which examines trends in the development of science in the CIS countries, and this article (in Georgian) focused on Georgian academic institutions. 

Junior Fellows at CRRC-Georgia: Facing new challenges

[Note: Over the next two weeks, Social Science in the Caucasus will publish the work of six young researchers who entered CRRC-Georgia’s Junior Fellowship Program (JFP) in February 2015.]

CRRC’s Junior Fellowship Program (JFP) was launched in 2009 as a Carnegie Corporation initiative within the CRRC, with the goal of providing on-the-job training opportunities in applied research for young social scientists. As the Program’s motto claims, the JFP would be “the best and hardest” experience the Junior Fellows would ever have, opening doors for educational and employment opportunities worldwide. Every year since, a new group of bright young people interested in social science research and selected through a highly competitive and challenging selection process would join CRRC-Georgia for a period of five to nine months.

Once becoming members of the CRRC-Georgia team, Junior Fellows are engaged in ongoing research projects at various stages of implementation both through direct participation and observation. Fellows learn about questionnaire development, pretest, sampling, survey fieldwork and back-checks, data management and analysis, slide production and presentation of results, report writing, the development of discussion guides, and preparation of interview and focus-group write-ups. We believe, Georgian higher educational institutions currently fail to help students acquire most of these skills, and hence the opportunity offered by CRRC-Georgia is quite unique for the Junior Fellows.

Potential junior fellows are expected to have already completed at least a BA degree, although it is not required that this degree is in the social sciences. They have to commit to the program full-time, and hence cannot combine JFP and another job or full-time studies during the fellowship period. In addition to valuable research experience, the Junior Fellows are provided with a number of training sessions on quantitative data analysis using SPSS and Stata statistical software, qualitative data analysis using NVivo software, and report writing.

Thirty two young people have been CRRC-Georgia Junior Fellows to date. After their fellowship period, seven stayed on to work at CRRC-Georgia, five continued their education abroad, and others started jobs at a number of different international and governmental organizations.

We asked some of the former fellows to share their opinions about what the JFP experience gave them, and how it relates to their current educational and employment situations. Here are some answers:

[At CRRC-Georgia] I got a valuable knowledge of how high-quality research should be conducted. While working with qualified people, I obtained knowledge and experience that helped me to advance in my future career. A great team, interesting projects and a great deal of perspectives – this is how I would describe my experience working at CRRC.
Nino Kerkadze, JFP-2012

CRRC is a small family, where one acquires new skills that will be useful [for the rest of your life].
Merab Bochoidze, JFP-2011

CRRC is a place with a wonderful working environment, qualified colleagues and great friends that will give a valuable boost to your career.
Salome Minesashvili, JFP-2011

CRRC-Georgia is currently hosting six junior fellows, and we believe, one of the most exciting, although challenging tasks for the current Junior Fellows is writing blog posts for the CRRC regional blog, Social Science in the Caucasus. In their blog posts, the Junior Fellows can focus on social science issues they are most interested in, apply their data analysis skills, and communicate their findings to an audience of international and local scholars, researchers, and journalists.

For the first time, we have decided to publish the 2015 Junior Fellows’ first blog posts in a single series. The JFP series will start today with Nino Zubashvili’s post and continue for two weeks.

CRRC Methodological Conference on Measuring Social Inequality in the South Caucasus and its Neighborhood

The second annual CRRC methodological conference took place on the 25th of June at Tbilisi State University. With over fifty attendees and a packed program of presentations, the conference drew together policy practitioners and researchers from the South Caucasus and beyond.

This year, for the first time, there was also a number of pre-conference workshops on the 24th of June at the CRRC-Georgia offices. Michael Robbins of the Arab Barometer presented on the matching techniques he has used to examine the Arab Spring, and Mihail Peleah from UNDP Europe and CIS presented an introduction to the methodology behind the UNDP’s Social Exclusion Index.

These provided an excellent introduction to the themes of social inequality in the South Caucasus and its wider neighborhood, also giving participants an opportunity to reflect on the implications of the methodological choices that we make on the results that we generate. There was a lively discussion about how to define and work with broad topics like social exclusion and inequality, and participants showed a keen interest in how these concepts had been applied in the Arab world, North Africa, Eastern Europe and Central Asia.

During the main conference, the geographic scope was expanded further to include in-depth studies on access to higher education, water subsidies and migration push factors in Armenia, inequality in educational achievement, local government performance, and domestic violence in Georgia, access to the benefits of a more technologically connected world in Azerbaijan, visual sociology in the post-soviet space, and data collection and visualization across the South Caucasus.

The broad geographic and methodological scope of the studies, as well as the high standard of papers received made this an excellent second edition to the CRRC series of methodological conferences in the South Caucasus. For more information, the full program and papers presented at the conference can be accessed here.

CRRC Methods Conference: Advancing Methodological Innovations in the South Caucasus

On June 20, 2013 CRRC convened a conference on research methods in Tbilisi that brought together researchers from the South Caucasus and beyond to discuss methodological developments in the social sciences. The conference was attended by over 50 people and the program included 13 presentations of research projects carried out in the region and elsewhere, followed by discussions and Q&A sessions. Interdisciplinary approaches and broad methodological interests motivated a fruitful debate on previous and ongoing research projects -both qualitative and quantitative. Time was also devoted to discussing the policy-relevance of presented studies, as well as the next steps that need to be taken in order to promote high caliber research and to foster rigorous methodological training in the region. 

The keynote speech, delivered by Dr. Cynthia Buckley, highlighted the institutional context in which social research is being carried out and results are disseminated. This included a discussion of funding organizations, policy advisory bodies, academic and social research institutes, and the complex network of relations between them. The speech helped participants to focus on methodological advancements, as well as practical applications of the presented studies. 

The research projects presented at the conference included a wide range of topics, research designs and used qualitative, quantitative or mixed-method approaches. Several presentations used online and internet-based methods – a promising medium for research in the region since the share of internet users is consistently increasing in the South Caucasus.

The qualitative studies that were discussed highlighted social issues such as interethnic marriage in the South Caucasus, and Armenian diaspora in the region. The quantitative research projects that were presented primarily focused on policy-relevant issues, such as the work ethic in Azerbaijan, Russia and Turkey, the prospect of compulsory healthcare insurance in Armenia, and time-based measures for paid and unpaid work, as well as labour force participation. Other areas of particular interest were business and the accuracy of economic performance estimations.

This 2013 conference begins a cycle of annual conferences that will address methodological challenges, data collection techniques, and developments in social science research carried out in the South Caucasus. 

For more information on the conference and this year’s presentations, please see the conference programme and draft papers available on the CRRC website. 

Third Stage of the Junior Research Fellowship Program at CRRC-Azerbaijan Launched!

In February 11, 2011, the CRRC-Azerbaijan office launched the third stage of its Junior Research Fellowship Program, funded by the Open Society Institute Think-Tank Fund. Fifteen selected participants will attend the next round of extensive trainings that will prepare them for writing public policy papers.

The major focus of the trainings this semester will be qualitative research methods. The participants will learn how to conduct in-depth interviews, focus groups and how to perform content analysis. They will also learn tips and tricks on how to transfer data from technical charts in SPSS to attractive and understandable graphs in PowerPoint. The semester will conclude with academic writing classes in which they will exercise making citations (footnotes and endnotes) as well as writing annotated bibliographies and literature reviews.

Throughout the third stage, new, interesting and challenging competitions will be organized. By the end of the course, the fellows will be armed with new skills and knowledge, as well as the inspiration to learn more about policy analysis and to become real professionals in the field.

Survey of PhD Students in Georgia

We recently undertook a small online survey of PhD students at Georgia's two major universities. This comes at a time when significant programs and support are already available to Georgian PhD students: CSS is launching a new PhD program, ASCN is offering significant research opportunities, the US Embassy will launch a program with Ilia State University, and now there is CARTI as a further opportunity.

The survey was done in Georgian, had 108 respondents, who probably are representative of active and engaged PhD students in Tbilisi, by virtue of responding to the request for participation. One should, of course, be cautious about generalizing from the results.

What, then, about existing students? 23 respondents said they had earned their degree abroad, while 75 said they had not. English seems in the ascendancy: 83 respondents said they had professional competency in English, compared with 66 in Russian, 12 in German and 6 in French.

PhD students are busy, and not only with their dissertation: 44 respondents said that they were teaching at university, and 81 respondents said they also had another job outside university. The jobs outside university are distributed across public-sector (33), NGO (25), private sector (20), and other (17). This illustrates that it may be difficult for students to focus on their research in the way that many Western PhD students can.

Libraries are surpassed by electronic resources. Only seven respondents say they use libraries. Free electronic materials are used by 31 respondents, and electronic catalogues such as EBSCO by 21, with 12 saying that they have a password to electronic libraries of universities abroad. Eleven respondents say they get materials from abroad. No one says that they use sources that exist in their department.

The upgrading of skills of Georgian professors at universities is seen as necessary or very necessary by 74 of the respondents. The PhD students themselves attend a fair amount of trainings. The last training they attended was on their field of specialization (28), teaching methods (26), research methods (22) and academic writing (5). 27 respondents said that this last training took place abroad, illustrating that PhD students enjoy reasonable levels of mobility.

And which skills do PhD students want to upgrade the most? 50 respondents told us they need training research methods, 21 want training in their particular field of specialization, 13 in teaching methods, and 10 in academic writing.

To be sure, this was the survey we organized in a little more than an afternoon, primarily out of curiosity. It suggests that more systematic work should be done to understand how to develop Georgia's research capacity. Given the amount of investment into PhD programs and research support, the PhD students themselves are curiously underresearched.

If you want access to the data, please post a comment or get in touch with us.

The CRRC Georgia Team

These are the CRRC Georgia team members who work hard on the numbers we usually present!

In this photo: celebrating the birthdays of three CRRC colleagues in Mtskheta with great food, great wine and charades in Georgian!

Ask CRRC: what does the public actually know?

A recent poll by the Pew Research Center showing that 18% of Americans think that US President Barack Obama is Muslim, and that a further 43% respond that they don't know what religion the President practices, has raised discussions about the level of political knowledge in democracies. Indeed, Newsweek has published a slideshow showing dumb things that Americans believe.

How do Georgians fare by comparison? Actually, Georgians seem remarkably well informed. In late 2007, in the run-up to the Presidential Elections, Jonathan K. suggested to us to include some questions in our pre-election survey (the entire effort was done in seven days, so even the slides don't quite match the standards we have today). As you can see, we had about 15% of respondents getting questions of national importance wrong. However, many respondents were willing to confess that they "don't know" the right answer, a healthy attitude we have also seen in other surveys.

Political knowledge 2007

View more presentations from crrccenters.

We know that public opinion is imprecise once we get into details: try and find out how Georgians look at detailed constitutional arrangements (sharing tax revenue between central and local government, Presidential versus Parliamentary system, that type of stuff) and you get the expression of a general desire for balance. Many respondents can't necessarily keep the Council of Europe and the European Union apart, but this is hardly surprising. Also, back when this was a topic, respondents struggled to distinguish NATO membership from the Membership Action Plan. In the end, we did find Georgian respondents so informed that we stopped asking knowledge questions.

According to more a survey conducted in October 2009, Georgians don't think that it's the bigger national and international issues they are not informed about.

It's things that happen in the other part of town that the media doesn't cover sufficiently.

Note one important difference between these opinion surveys: the American polls mostly are conducted on the phone, and may have a bored respondent paying insufficient attention. Moreover, the polls with surprising results capture a disproportionate amount of the attention. So one should not overstate the US results. The Georgian surveys are conducted face-to-face, and respondents are more likely to struggle to find the right answer. 

More questions? Let us know.

Ask CRRC!

When presenting our work, or talking about it informally, we are asked fairly similar questions: do you do your interviewing in all of the country? How do you select the respondents? How do you know they are not lying to you? Are people willing to say things critical of the government? How do you design a questionnaire?

These are extremely important questions, because they will influence whether you can take our survey results at face value. As mentioned in the last post, we have decided to give you more regular updates on what we do, and how we do it.

This, too, was another lesson we learned from our favorite role models, the Pew Research Centers. They have a specific section called "Ask the Expert", pictured below.

What, then, has always puzzled you about survey research? Let us know, either through the comments or by writing an e-mail. We will, eventually, make this information available in the local languages as well. Your input will help us identify the questions people have.

So, what questions do you have for us?

More News & Numbers from CRRC

Over the last few weeks and months, we have regularly posted updates about what's going on, and where we stumbled on information we thought was interesting. We think these are useful contributions: small snippets, searchable, easy to find through Google, and a way for us at CRRC to think about synthesizing complex research into a handful of paragraphs. Note some of the emerging themes, such as the question of life satisfaction.

What we have done less systematically is to give you a glimpse about what's going on within CRRC. We will change that, by giving you more updates on work we are doing, people we are working with, and what they're doing. We also want to use this opportunity to set out more clearly why we think that the work we're doing is important, and is a good model of development. For us, it's not one project after another, although it may sometimes feel that way. There's a rationale that we would like to share with a broader audience.

Senator Patrick Moynihan coined the phrase that "Everyone is entitled to his own opinion, but not to his own facts." Essentially, what CRRC seeks to contribute is facts, so that people can have more than just an opinion. That, in a nutshell, is what quality research in a transition context should be about.

Put us into your CSS feed for more regular updates, and consider subscribing to our new e-bulletin, relaunched as News & Numbers from CRRC. You find details here.

CRIA: 2009 Autumn Issue

The Caucasian Review of International Affairs’ (CRIA) Autumn issue has arrived.

Since 2006, the non-profit, quarterly academic journal has been publishing works from a wide array of international scholars, analysts, and researchers. Committed to providing a better understanding of regional affairs, the CRIA is unique as a free, peer-reviewed online academic journal devoted to covering the South Caucasus.

In the interest of promoting an exchange of ideas and dialogue on this fascinating part of the world, the CRIA publishes papers, comments, book reviews, and interviews, as well as its weekly Caucasus Update, all of which provide in-depth analysis on affairs in the Caucasus as well as the wider region.

Representing several different academic institutions, the CRIA’s international advisory and editorial boards lend their expertise and experience to the journal, and its readership continues to grow. Further, the CRIA was recently added to Columbia International Affairs Online, and is now included on a large list of international citation indexes and research databases, and in numerous universities’ e-journal catalogues. Several mutually beneficial partnerships have been established as well, including one with the CRRC.

Kartvelophiles will find plenty to pique their interest. The headline paper for the Summer ’09 issue analyzes patterns of balance and bias in several international newspapers’ coverage of the 2008 Russia–Georgia war. The current autumn issue includes a paper by Alexi Gugushvili on the reform of the old-age pension system in Georgia and an interview with CRRC’s Regional Director Dr. Hans Gutbrod and its Georgia Country Director Koba Turmanidze.

And do not forget to browse the back issues, too, and check out Aaron Erlich’s review of Magnarella’s “The Peasant Venture” for a fascinating look at a work that goes beyond standard political and economic themes. In addition, other noteworthy pieces by Dr. Papava of the GFSIS, Lasha Tchantouridze, and Till Bruckner’s paper on the government’s efforts to house IDPs can also be found in the back issues.

Finally, for all who are interested, the CRIA accepts papers, comments, and book reviews on a rolling basis (see our submission guidelines for further details), and all manuscripts are carefully considered. Submission deadlines for the Winter 2010 and the Spring 2010 issues are December 15, 2009, and March 15, 2010, respectively. Feel free to e-mail www.cria-online.org with any questions or comments.

New Google Squared – a useful research tool?

On 9 October, Google introduced a number of improvements to its search tool “Squared”. Squared was first presented in May 2009 with the idea that instead of going through a big amount of WebPages, the new search tool would provide for a collection of facts presented in tables of items and attributes, which is what Google refers to as “squares”. Google says that Squared is a helpful tool when you are searching for more complex information that the normal Google search tool cannot manage and you need to visit several WebPages in order to collect all the material needed. The result is similar to a spreadsheet, and you are able to see the websites that serve as sources for the information in the squares.

However, when Squared was launched the initial reactions were mostly negative. The main critique was that the results were rather irrational and illogical. Several improvements have thus been made now. More squares with information can be included, and according to Google, the quality of information has improved and is ranked based on relevance and whether high quality facts are available. Data can now also be exported to Google Spreadsheet or a CSV file. Additionally, Squared is re-designed to learn from edits and corrections of its users.

So how well are these improvements working out? And can Google Squared be useful for Caucasus-related research? Unfortunately, Squared is still a limited search tool in several aspects. The basic idea of Squared is sound and could probably come in handy for students of intermediary stages of research, or, to take an example that Google uses, to find out different information about US presidents. As an advanced research tool, however, it is still not entirely adequate. For example, when searching for Scandinavian countries you are provided with some basic information ranging from language, way of governance, GDP per capita and the number of Internet users. Indeed, this provides for an overview and comparison. A similar search for the Caucasian countries does not provide for an as useful overview, though. The information is scarce and there are not a variety of sources either, as the absolute majority of information squares derives from Wikipedia. Moreover, for many of the attributes there are no values found, such as for unemployment rates and information about national industries. The recent improvements to Squared are thus not a real breakthrough yet. Also, quite surprisingly, English is listed as the preferred language is all three countries. If Squared would thus be used by someone with little knowledge about the Caucasus, it would give a slightly misleading picture.

All in all, very little information is to be found about any topics on the Caucasus. For more advanced purposes and social science research related to Caucasus, it is simply not a useful tool. In comparison to the usual Google search and Google Scholar, it is difficult to see the additional advantages and usefulness that Squared would bring. Google points out that the program is only in its experimental stage, and it remains to be seen if a person wishing to deepen their knowledge about different topics in the Caucasus could gain from Google Squared in the future.

"Is Georgia a Democracy?" | Recent Publication

Is Georgia a democracy? In previous blog posts we tracked various indicators, including the Freedom House Index. But what do Georgians themselves think?

Koba Turmanidze, Director of CRRC Georgia, and Hans Gutbrod from the CRRC Regional Office have written a short chapter discussing poll findings on this question. It is part of a broader publication by the Foreign Policy Centre, a UK Think Tank.


The publication also includes essays by Peter Semneby (EU Special Representative), Giorgi Gogia (Human Rights Watch) and Giorgi Chkheidze (Georgian Young Lawyers/Ombudsman's office). It also has fascinating electoral maps that we meshed up for NDI.



To read, click here.

Google Insight | Search What the World Searches

We have been consistently impressed over the last few years how great research has become possible with simply an Internet connection. Google Scholar and Google Books have added new dimensions, so much so that the Resource part in our name has become a lot less relevant than it used to be. Now our task often is pointing scholars to the right search engine, rather than providing basic access.

A recent addition is Google Insight, the "search of searches", i.e. finding out what other people are looking for and when and where and how.

Take a look at this overview (you may want to maximize your screen), telling you how this great tool works in less than five minutes.




As we point out, there are some limitations across various languages, since search terms will differ. Nevertheless, it's a remarkable tool.

Try it out with your favorite search terms on the Google Insight webpage.

Announcing New Fellowship | UC Berkeley Scholars Workshop

The Institute of Slavic, East European, and Eurasian Studies (ISEEES) at the University of California, Berkeley (UCB), in collaboration with the Caucasus Research Resource Centers (CRRC), is soliciting proposals from scholars in the Social Sciences and History from Armenia, Azerbaijan, and Georgia to participate in a two-week workshop at UCB from November 7 to November 21, 2009. The workshop is funded by a generous grant from the Carnegie Corporation of New York.

A total of four scholars from the South Caucasus (“Carnegie Fellows”) will be brought to UCB for an intensive review of the key literature, theoretical approaches, and methods employed in a particular field of scholarship. Each Carnegie Fellow will work with a paired UCB faculty member and graduate student with knowledge of the Carnegie Fellow’s field theme to develop undergraduate and graduate syllabi and teaching materials, explore innovative teaching and research techniques and technologies, and prepare a literature review for use by other Carnegie Fellows and scholars from the South Caucasus. The language of the workshop will be English.

Airfare, hotel, and meal expenses will be paid for by ISEEES. In addition, ISEEES will either pay for or reimburse each Carnegie Scholar for up to $600 in expenses relating to purchasing, copying, and posting teaching materials. ISEEES will provide letters of invitation, but each Carnegie Fellow will be responsible for obtaining a US visa with assistance from the local CRRC offices in Baku, Tbilisi, and Yerevan.

Who is eligible to participate?

• Citizens of Armenia, Azerbaijan, and Georgia
• Scholars who hold a "Kandidatskaya" degree or higher
• Scholars who have a level of proficiency in written and spoken English that is sufficient to conduct independent research and engage in a debate.

How to apply/documents to submit

• Completed application form
• Statement of purpose, explaining the applicant’s research and teaching experience and future plans (not exceeding 3 pages)
• Sample of scholarly research (not exceeding 10 pages)
• Curriculum Vitae

Application forms can be found at CRRC website. The website also has a detailed description of the purpose and design of the workshop. Applicants should be sure to read the Program Description to ensure that they are familiar with the design and requirements of the program.

Applications should be submitted via email to applications@crrccenters.org no later than September 11, 2009.

Carnegie Research Fellowship Program!

CRRC is happy to announce the Carnegie Research Fellowship Program (CRFP). The program offers exceptional research opportunities in the United States for scholars from the South Caucasus.

Specifically, scholars in the social sciences and the humanities may apply for individual, non-degree research opportunities at universities and institutes in the United States. The program is directed at advanced researchers that already have a demonstrated track record in research. The research period lasts up to a full semester (4 months), starting either September 2009 or January 2010. In 2008-2009, two fellows from Georgia and one from Armenia have been sent to Harvard University, University of Chicago and University of Washington to do their research.

Individuals who are eligible to participate in the fellowship program:

• Citizens of Armenia, Azerbaijan, and Georgia.
• Advanced graduate students, university faculty and scholars at any stage in their careers who have not recently conducted research projects at U.S. institutions.
• Scholars who hold a “Kandidatskaya” degree or higher, or who are working towards a “Kandidatskaya” degree at the time of application.
• Scholars who have publications (advanced graduate students may cite papers presented at academic conferences) in a particular field.
• Scholars who have a level of proficiency in written and spoken English that is sufficient to conduct independent research and engage colleagues.
• Scholars who are able to receive and maintain a United States J-1 visa.
• Scholars who are able to begin the CRFP in the United States in September 2009, or January 2010.
  

NCEEER, the American Councils, and the CRRC do not discriminate on the basis of race, colour, sex, religion, national or ethnic origin, and disability. For more information please visit NCEEER website.

Applications need to be submitted in a hard copy to your local CRRC office. Deadline for applications is April 30, 5 p.m., 2009. We suggest applicants to study details in the guidelines and the application form closely, and in good time, to avoid disappointment. We will be accepting applications in the social sciences and the humanities. All costs for the scholars are covered, including round-trip airfare.

CRF

View more presentations from crrccenters.

The Carnegie Research Fellowship presents an extraordinary chance to researchers that can advance their work through a period of self-directed study in the US. Note that the application process is very competitive, since a concise research proposal is expected.

In order to get application materials, go to CRRC website. If you are interested in getting further training on how to improve your application, please email nana+nceeer@crrccenters.org with 'interactive online trainings' in the subject line.

EBRD Life in Transition Survey | worth analyzing!

One of the most impressive recent survey efforts, measuring attitudes about different countries in transition, has been undertaken by EBRD. Called Life in Transition Survey (LiTS), this is an attempt to look at how 29 'transitioning' countries have developed following 1989. The survey tracks "public attitudes, well-being, and the impact of economic and political change".

LiTS is both encouraging and sobering at the same time. What is encouraging is that overall people, in spite of many hardships, do not want a return to centralized, authoritarian systems. At the same time, incomplete transition has left many people equivocal with regards to market systems. EBRD also notes that social capital remains in short supply.

The basic idea is set out succinctly in this EBRD presentation.

Life in Transition 2

View SlideShare presentation or Upload your own. (tags: ebrd economy)

(Note that we really think such presentations should be available under a Creative Commons license, since technically we are not sure whether we're allowed to share this, although it's available online.)

But beyond this, LiTS offers much more, and much more detail on the Caucasus. Let's look at some of the findings in more detail.

1. Satisfaction with life is relatively low in Armenia, and the old and poor tend to be most dissatisfied. Only 20% of those aged over 50 are satisfied with their lives. And among the poor, the situation is even worse: only 16% of those with lower income are satisfied with their lives. The situation is similar in Georgia, where only 12% of those aged 50-64 are satisfied with their life. By contrast, satisfaction with the economic and political situation is relatively strong in Azerbaijan, particularly among the 50- 64 age group. This seems to tally with what we saw in our own 2007 Data Initiative (although interesting to compare and contrast with Elvin Effendi's work).
2. In Armenia support for democracy and a market economy is weak, with just one in four favoring a combination of the two. In Georgia, there are high levels of support for democracy, but less for a market economy, with those aged over 65 most strongly opposed to both (over 50%). In Azerbaijan, support for democracy and a market economy is high, with the middle-aged the most supportive. However, there also is little interest in the political and economic system, with four out of ten believing that the type of political/economic system does not matter.
3. Azerbaijanis' trust in public institutions, especially in the presidency, government and the political parties is among the highest in the region. By contrast, trust in public institutions is very low in Armenia. The armed forces is the only public institution in Armenia that enjoys public trust.
4. Georgians report a decline of corruption in the country and the frequency of “irregular payments” to the public officials is significantly lower than elsewhere in the CIS regions. “Irregular payments”, especially in the healthcare and education spheres, remain relatively high both in Armenia and Azerbaijan.
5. In all three South Caucasus countries people surveyed are more optimistic about the future of their children. The optimism is significantly higher in Georgia, where almost 70% of those surveyed believe that their children’s lives will be better than their own.

Separately, we are really impressed by this entire effort. It is a genuine public good. EBRD is frank about discussing shortcomings and challenges in the societies (and therefore note that the views are those of the Office of the Chief Economist, not necessarily of the entire Bank). This is incredibly refreshing in an environment where blandness prevails. The data set is publicly accessible, there's detailed documentation about questionnaire, fieldwork, methodology. And most of the material is available online.

The full printed report (soon in our libraries, if you find it too expensive) is written so well that it makes engaging bedtime reading. Our only huge regret is that these discussions seem not to have been carried into the relevant societies. If any donor is short of creative ideas, surely this is a way to go: let's get this data studied and analyzed locally, and let's get the discussions onto TV. (If anyone needs help with the dataset, let us know, or come to our offices.)