Respondent Evaluation | A Great Tool for Looking into Survey Interviews

What are the patterns in how the respondents are rated by the interviewers? The relevance of this question is beyond doubt, as patterns in such ratings allow for an idea of the reliability of the data as well as for more general insights into the settings in which interviewers are gathering data. Relevant data has been gathered in the Caucasus Barometer (CB) survey for years, enabling us to analyze the impressions that interviewers have gained during their work in the South Caucasus.

One particularly interesting endeavour is to make comparisons between the perceptions of interviewers in the three countries – Azerbaijan, Armenia and Georgia. In all three cases, interviewers were asked to evaluate issues affecting the interview such as levels of distraction, frequency of the need for clarifications, or the extent to which respondents were reluctant to answer questions. Perceptions of the personal characteristics of respondents, such as sincerity, comfort and interest, are also recorded. Patterns of interviewer’s ratings can subsequently be identified by looking at several of the measured variables. These also allow us to filter out interviews in which respondents were uncomfortable, or seemed insincere.

One first interesting finding is that interviewers in Azerbaijan seem to face more obstacles in the interviewing process when compared with their counterparts in Armenia or Georgia. More concretely, respondents in Azerbaijan are more frequently in need of clarifications of questions and also show more often a lack of knowledge on a certain subject. Only in slightly above 20 percent of the cases were the interviewers in Azerbaijan ‘never’ (i.e. not throughout the entire interview) confronted with such situations. By contrast, in the neighbouring countries, such instances comprise over 30 percent (see Figures 1 and 2). A comparison between Armenia and Georgia shows that inhabitants of Georgia seem to have difficulties in slightly fewer instances, and in particular appear more knowledgeable to the interviewers. (Interestingly, there is some evidence that respondents in all countries seem to face more difficulties in answering questions without clarifications in 2009 as compared to 2007-08, but this is yet another topic.)

Given the above findings, it is then arguably strange that Azerbaijani and Armenian respondents were labelled as "more comfortable" compared with the Georgian respondents (see Figure 3). In other words, although Georgians are perceived by their interviewers to have less trouble during the interview (as reflected in need of clarifications and lack of knowledge), this does not coincide with a more comfortable behaviour even if one might have expected that. However, in all three instances, there is a comparably low number of nervous respondents (slightly less than 10 percent), which suggests that nervousness is not a particularly serious problem in any of the surveys.

Finally, of the many interviewer ratings the CB includes, another remarkable one is the evaluation of interest and involvement. The first striking aspect here is that in all three countries, most respondents do not appear to be interested or involved during the interview. Moreover, Georgians are least interested/involved in the interview process (less than 30 percent were assessed as interested), whereas Azerbaijanis are the most interested (approximately 45 percent; see Figure 4). Armenian subjects are, as in the above cases, perceived to be somewhere in the middle (40 percent). Thus, while interviewers in Azerbaijan seem to be more often confronted with a lack of knowledge and confusion about questions, interviewers in Georgia more often come across higher levels of perceived disinterest and lack of comfort on the part of the their respondents.

 

To be sure, these ratings are done at the end of the interview, and we know that we cannot ask complex questions at this point anymore. Typically we've been talking for 45 minutes and have exhausted the patience of the respondent. Nevertheless, these findings remind us that we need to keep our questionnaires short.

While we know that particular wordings may lead to slightly different interpretations in different languages, the interviewer ratings give us a glimpse into what is going on, and thus help us improve the quality further. Survey work is nearly never perfect. It's a process of continuous improvement. Plenty of opportunities for researchers to analyze the data more closely.