Developing the “culture of polling” in Georgia (Part 2): The misinterpretation and misuse of survey data

[Note: This is the second part of a guest blog post from Natia Mestvirishvili, a Researcher at International Centre for Migration Policy Development (ICMPD) and a former Senior Researcher at CRRC-Georgia. The first part of this blog post is available here. This post was co-published with the Clarion]

The misinterpretation of survey findings is a rather widespread problem in Georgia. Unfortunately, it often leads to the misuse of data, which not only diminishes the importance of survey research, but also leads to more serious consequences for the country.

To illustrate how one might misinterpret survey data, the following example from CRRC’s 2015 Caucasus Barometer survey can be used. When asked, “What do you think is the most important issue facing Georgia at the moment?”, only 3% of the population mentioned low pensions, 2% the unaffordability of healthcare, and 2% the low quality of education. A number of issues including the violation of human rights, unfairness of courts, corruption, unfairness of elections, unaffordability of professional or higher education, the violation of property rights, gender inequality, religious intolerance and emigration were grouped into the category “Other”, because, in total, only 7% of the population mentioned these issues.

Based on these findings, one might think that these issues are unimportant in Georgia. However, this would be a misinterpretation, which happens for a number of reasons. Here, I focus on two. The first is:

1. Not paying attention to the exact formulation (wording) of the survey question, answer options, and instructions 

One reason a large share of the population did not mention the violation of human rights, gender inequality and religious intolerance as important issues is because each respondent could name only one issue. The options they chose (unemployment and poverty were named most often) were more important to them than human rights, gender inequality, and religious intolerance.

If a different question – “How important is the issue of human rights [or gender inequality, or religious intolerance] for Georgia?” – had been asked, the share of people who would answer that these issues are important would very likely be much higher than one or two percent. This wording would make people judge the issue not in relative, but absolute terms.

While working with survey findings, the exact wording of question(s) should always be taken into account. When the question is interpreted or reworded, it will almost inevitably lead to some degree of misinterpretation. More often than not, fieldwork instructions should also be taken into account. For example, was a show card used for the question? Was the number of answer options a respondent could choose limited or not?

Thus, it is crucial that survey results are understood and reported, keeping in mind the exact wording of the question(s), answer options provided, and any instruction(s) that had to be followed during the interviews. This will help minimize the risk of misinterpretation.

A second common cause of misinterpretation of public opinion polls in Georgia is:

2. Interpreting public opinion survey results as ‘reality’ rather than perceptions 

Even if the question discussed above had been asked so that the absolute rather than relative importance of the issues was measured and the survey findings still suggested that people thought the violation of human rights, gender inequality and religious intolerance were not important issues for the country, the findings should not be interpreted as a direct reflection of ‘reality.’ As discussed in the first part of this blog post, public perceptions are not ‘reality’.

Interpreting public perceptions as objective ‘reality’ is incorrect, because both perceptions and misperceptions, information and misinformation shape public opinion. It is equally important to remember that, sometimes, ‘reality’ simply does not exist. Moreover, as a number of studies have shown, it is often the case that people are simply wrong about a wide variety of things.

None of the above, however, diminishes the role and importance of public opinion polls. In fact, the misperceptions that survey findings can uncover are often among the most important outcomes for policymakers. Instead of putting an equal sign between public perceptions and ‘reality,’ data analysts and policymakers should critically analyze and address gaps between the two.

Going back to the above example, an accurate interpretation would consider the findings in the context of other studies that are specifically focused on human rights (or gender equality or religious tolerance). Indeed, numerous studies indicate that Georgia has serious problems with all three issues i.e., the population does not have much respect for human rights, gender equality, or people of other religions. Only looking at the latest Human Rights Watch report on Georgia makes this quite clear.

Looking at inconsistencies between people’s answers to different questions, or between survey findings and other types of data when available and relevant, is a good way to uncover misperceptions. For example, a 2014 CRRC/NDI survey found that roughly every fourth person reports there is gender equality in Georgia. However, about half of those who think so also think that taking care of the home and family makes women as satisfied as having a paid job, and that in order to preserve the family, the wife should endure a lot from her spouse.

The answers to these three questions should be presented and discussed not separately, as independent findings, but rather as interrelated findings that, taken together, give a better understanding of the assessments of and attitudes towards gender equality in Georgia. In this context, the question that needs to be raised and answered is why and how this inconsistency between answers occurs.

The misuse of survey findings happens when findings are presented and used in a way that reinforces people’s misperceptions and prejudices. The misinterpretation of findings often leads to their misuse, and eventually, can lead to serious issues.

Again, going back to the most important issue example, it would be a misuse of survey findings to conclude that since the violation of human rights, religious intolerance or gender inequality seem to not be perceived as important issues in Georgia, no policy is needed to address them. As demonstrated above, alternative sources show that these issues need to be addressed, and, at the very least, awareness of them needs to increase. Thus, policy intervention is needed.

What the survey findings tell us in this case is that people underestimate the importance of these issues. In turn, this contributes to the worsening of the problems. If you believe gender inequality or religious intolerance are not important, you probably would not care about these issues either. Thus, the larger is the gap between public perceptions and reality, the more important it is for policy makers to address the issue.

Public opinion should not be used as a directive for policy making without careful analysis of misperceptions and alternative sources of information.

Unfortunately, in Georgia sometimes it’s exactly the misperceptions that drive policy. Speaking of recent developments, misperceptions about homosexuality have lead politicians to talk more about the prohibition of same-sex marriage, something that has never been allowed in Georgia in the first place, than about human rights issues. Misperceptions about gender roles led politicians to reject a proposal that would define femicide as a premeditated murder of a woman based on her gender. Looking forward, the country cannot allow the misperception that the EU threatens Georgia’s traditions to drive the country’s foreign policy.

Now more than ever, when Georgia is still attempting to transition into a stable, democratic country, the country needs policymakers and researchers who have the knowledge and skills to critically analyze survey findings and use their potential for the development of the country.