Who is more intelligent and sincere? CRRC interviewers’ assessments

It has been only about two decades since pollsters started using paradata, including interviewer assessments of the conducted interviews and respondents’ behavior and attitudes. Such assessments are collected in the process of most of CRRC-Georgia’s surveys. Immediately after each interview, interviewers are asked to assess various aspects of the interview, including respondents’ sincerity and intelligence. Using CRRC’s 2015 Caucasus Barometer (CB) data for Armenia and Georgia, this blog post explores how such assessments differ depending on respondents’ demographic characteristics and their households’ economic situations.

It is important to highlight that these assessments can hardly provide reliable information about the actual intelligence or sincerity of respondents, and there is no doubt that interviewers’ perceptions of intelligence are very subjective, thus they cannot be attributed to any objective characteristics. Still, when analyzed properly, they can tell us a lot about a wide range of issues, from the process of interviews to the dominant stereotypes and biases in a given society.

In the CB 2015 assessment forms, interviewers were asked to rate respondents’ intelligence. In both Armenia and Georgia, interviewers tend to rate the intelligence of people who live in the capital and other urban settlements higher than the intelligence of those who live in villages.

 Note: The original 5-point scale was recoded for the charts in this blog post, with answer options “Intelligent” and “Very intelligent” combined into the category “Intelligent” and answer options “Not at all intelligent” and “Not very intelligent” combined into the category “Not intelligent”. Responses “Don’t know” and “Refuse to answer” were excluded from the analysis in the charts in this blog post.

Although, overall the respondents are reported to be rather sincere during the interviews, interviewers report the answers of people living in the capital and other urban settlements to be slightly more sincere compared to the answers of people living in villages.

Note: The original 11-point scale was recoded for the charts in this blog post, with codes 0 through 3 combined into the category “Not sincere”, codes 4 through 6 combined into “Average” and codes 7 through 10 combined into the category “Sincere”. 

There are even more pronounced differences by respondents’ level of education. Interviewers rate people with a lower level of formal education both as less intelligent and sincere. In both countries, people who report having secondary technical education or secondary or lower education are twice as likely to be placed in the category “Average” than those with a higher than secondary education.

 Note: Answer options “No primary education”, “Primary education”, “Incomplete secondary education” and “Secondary education” were grouped into the category “Secondary or lower”; “Incomplete higher education”, “Completed higher education” and “Post-graduate degree” were grouped into the category “Higher than secondary”. 

In contrast to the assessments of intelligence, interviewers’ assessments of respondents’ sincerity vary less by the respondents’ level of education. Both these assessments, however, are very similar in Armenia and Georgia.

Interviewers’ assessments of respondents’ intelligence and sincerity also differ according to the reported economic situation of the respondent’s household. Interviewers tend to perceive those who are better-off economically as more intelligent. For example, interviewers report people living in households that can afford to buy expensive durables to be intelligent almost three times more often than people who do not have enough money for food, in both Armenia and Georgia.

Note: Answer options “We can afford to buy some expensive durables [like a refrigerator or washing machine]” and “We can afford to buy anything we need” were grouped into the category “Enough money for expensive durables”.

Those who report being in the best economic situation tend to be rated as sincere more often than people in all other groups.

Further research is needed to find out to what extent interviewer assessments reflect the actual situation, i.e. whether those who are better off, or urban dwellers, are actually more intelligent or sincere, and vice versa. The findings presented in this blog post show that interviewers have rather strong opinions about what groups of respondents are intelligent and sincere, and the assessments are very similar in Armenia and Georgia. Interestingly, there is a high positive correlation between the assessments of respondents’ intelligence and sincerity in both countries, suggesting another interesting topic for further research, namely, to what extent intelligence and sincerity are seen by interviewers as interrelated characteristics.

To find out more about CRRC’s Caucasus Barometer and other surveys, check out our online data analysis tool.

What do CB interviewers’ ratings of respondents’ intelligence tell us?

CRRC’s Caucasus Barometer (CB) surveys regularly collect information about how the interviewers assess each of the conducted interviews – so called paradata that provides additional insight into the conditions surrounding the interviews (e.g., whether someone besides the respondent and the interviewer was present during the face-to-face interview), as well as interviewers’ subjective assessments of, for example, level of sincerity of the respondents. This type of knowledge is especially important not only because the populations of the South Caucasus countries often distrust polling and pollsters (Hayes et al., 2006), but also because the conditions during the face-to-face interviews are rarely perfect in the region. The latter usually stems from objective reasons, like, for example, crowded dwellings (especially in the winter, when families normally heat only a few rooms, where all household members tend to gather). This blog post looks at the information provided by CB 2013 Interviewer Assessment Forms in Armenia, Azerbaijan and Georgia, trying to determine whether there is a bias in interviewers’ assessments. Specifically, we will have a look at interviewers’ ratings of respondents’ intelligence.

Overall, CB interviewers tend to rate the majority of respondents as average to (moderately) intelligent, but rarely as “very intelligent”. Interestingly, Georgian interviewers’ ratings are more skewed towards the positive edge of the scale, compared to those in Armenia and Azerbaijan.

Note: The data was not weighted for the analysis performed for this blog post. 

These figures, of course, in no way provide objective assessments of the intelligence of the respondents – neither was it the aim of this exercise. We will now look at how these assessments correlate with respondents’ answers on their basic socio-economic characteristics, including gender, age and level of education.

In all countries, there is a strong positive correlation between interviewers’ assessment of intelligence of the respondents and the two variables measuring respondents’ level of education - highest level of education achieved by the respondent, and the years s/he spent in formal education. The strength of correlation is somewhat stronger in Azerbaijan (.549 for the first variable) and somewhat weaker in Georgia (.455) and Armenia (.468).

Azerbaijan is the only country where, according to CB 2013, there is a relatively weak, but significant correlation between interviewers’ assessment of respondents’ intelligence and the respondents’ gender (-.192), suggesting that there is a systematic tendency to rate male respondents’ intelligence higher than that of females’. This finding is all the more interesting since a bit over half of CB2013 interviewers in Azerbaijan were female (22 male interviewers, 24 female interviewers). In all countries, there is a tendency to rate the intelligence of those living in the capital cities higher than those living in other settlements – the respective correlation coefficients are not high (-.094 in Azerbaijan, -.119 in Georgia and -.155 in Armenia), but are significant. There is, however, no correlation between these assessments and respondents’ age.

Interviewers’ ratings of respondents’ intelligence do not seem to be straightforwardly related to the attitudes towards democracy, or reported trust towards major social and political institutions, although respective findings differ by countries. It’s only in Georgia that we can see weak, but negative correlation between trust towards the president of the country and assessment of respondent’s intelligence by interviewer (-.141), while the correlations between these variables are not significant in Armenia and Azerbaijan.  

What other variables might, in your opinion, affect an interviewer’s assessment of a respondent’s intelligence?

You can have a look at CB’s Interviewer Assessment Form at the end of each CB questionnaire, e.g., here; and, of course, you can learn all about CB data on CRRC’s Online Data Analysis tool.