Does our algorithm still work?

Within the Russian Propaganda Barometer Project, funded by USAID through EWMI’s ACCESS program, CRRC-Georgia created a model, using a k-nearest neighbors algorithm, which attempts to predict whether a person falls into one of three groups: consistently pro-Western; anti-Western; or neither and potentially at-risk of being influenced towards an anti-Western foreign policy position. The model used data from NDI and CRRC’s polling between 2008 and July 2018.  It included variables for age, education level, settlement type, and when the survey was conducted.

In essence, this was done to see whether it is possible to guess a person’s foreign policy preferences using the aforementioned variables. The model successfully predicted people’s status 64% of the time, as described in this policy brief. This was a nine percentage point improvement over always guessing the most common status.

But, as is well known, social scientists can play around with data to make themselves look good. In this regard, one way of understanding how well a model works is whether it can predict observations that are inaccessible to the researcher at the time of the development of the model i.e. does the model make accurate predictions about people it does not yet have information about?

To test the model, CRRC-Georgia used the same algorithm to predict people’s status on the December 2018 and April 2019 NDI survey. Neither dataset was available to the researcher at the time the model was developed.

The results suggest the model works in near identical form, predicting 65% of responses accurately. As in the policy brief, the new data suggest that people in predominantly ethnic minority settlements, people with lower levels of education, and older people are significantly more likely to be at risk of being influenced by anti-Western propaganda in Georgia.


These results lead to a number of conclusions. First, the model does appear to work. In essence, it can guess someone’s foreign policy position correctly two thirds of the time if you know the type of settlement they live in, age, and education level.  Second, it re-affirms the recommendation in the policy brief that people working towards countering anti-Western propaganda in Georgia should prioritize working with ethnic minority communities; people who are not so young; and those with lower levels of education. At some level, these groups are relatively difficult for NGOs to reach. NGOs often focus on working with young people and the highly educated. Moreover, NGOs rarely have the capacity to work in Armenian and Azeri communities, aside from the organizations from those communities.

Despite the challenge in reaching these populations, the need is clear, and if Georgia is to prevent anti-Western propaganda from dividing society actors should work to counter Russian propaganda efforts among these groups.

The views expressed in this blog post do not represent the views of EWMI, USAID, or any related entity.

Replication code for the above data analysis is available here.