ECFR report: Befuddling data

Public opinion found its way into a major report by the European Council on Foreign Relations (ECFR), but through the back door. The chart below, from page 28 of the report, appears to compare support for integration with Russia/CIS versus EU integration in the six EU “neighborhood” states.


But the footnote reveals that this data is a pastiche from a number of national opinion surveys that asked questions about attitudes toward EU integration. This type of data presentation can lead us astray, for a few reasons:

1. Comparability: A footnote claims that the variously worded questions are nevertheless “roughly comparable.” However, the subjects of the questions range from actual political integration, to foreign policy alignment, to “strategic partnership.” Some are concrete (“If a referendum were held next Sunday…”), others abstract (“With which of the following does Armenia’s future most lie?”). The form of the questions also varies. Some explicitly offer a choice between Russia and EU, others probe attitudes about the EU alone, still others offer unknown options for partnership. More fundamentally, many respondents in the “neighborhood” countries may not believe that EU integration is actually a feasible option. Asking about preferences for integration in the CIS versus the EU is meaningful only if people feel this is a realistic choice.

2. Unknown Sources: There is no indication of the survey sources. Even if the data itself is of high quality, methodology certainly was different in each of the polls. Timing is a particular concern. Commendably, the authors of the report do note that the survey in Georgia was carried out in 2007, while the others were in 2008. But political events at various points during those years (the Russia-Georgia conflict, the economic crisis, gas disputes between Russia and Ukraine, among others) could influence responses.

3. Presentation: For some countries, the combined responses total nearly 100% (Belarus, Ukraine), others are far less (Georgia, at around 50%) or far more (Moldova, with 120%). Presumably this reflects the different types of questions asked, or possibly missing values. But the chart fails to tell us which responses account for these discrepancies.

This kind of data presentation is a little disconcerting. Although it is very encouraging to see public opinion data in a major report, one would wish for a slightly more cautious presentation. To be able to draw powerful conclusions, a more consistent approach to gathering the data would be required.

In the coming days, we’ll put up a follow-up post presenting CRRC’s data on attitudes toward cooperation with EU and Russia in the South Caucasus countries.