Pre-Election Polls | what would be needed

With the election in Georgia approaching fast, polls are beginning to appear every week. Unfortunately, many of these polls are taken at face value. The reality is that at this point there is not a single pre-election poll that has demonstrated credibility. This does not necessarily mean that polling firms and newspapers are simply fabricating their data -- it simply means that if they were simply fabricating their data, it would be very difficult for anyone to know.

So can we be confident that a poll is credible? There are a number of basic stipulations:

1. Reveal the sampling methodology. How, in other words, do the pollsters ensure that interviewing a few thousand people is representative of the entire electorate? Choosing respondents requires a) knowing where most people live, and b) having a very strong theory about which people are likely to turn out to vote on election day. This is very difficult stuff, and even tiny errors here can have tremendous consequences.

2. Tell us about the field work. Were the interviews done face to face or by telephone? When and how? Did the survey enumerators explain who they were working for, and is it possible that the respondents knew that they were looking certain answers?

3. Publish the questionnaire. What exactly was asked, and how, and in what sequence?

4. Document the non-response rate. How many people refused to answer? There are plenty of people who don't pick up the phone, or who don't have 30 minutes to talk to pollsters...and in this country, many of those people will vote.

5. Allow peer-review. Power point presentations for nonspecialists are fine, but make the data set available to peers for professional scrutiny (and of course you can restrict usage). If you really are confident in what you're doing, this is the way to go.

If polls do not meet the standards, they really do not deserve to be taken seriously.

Too many commentators forget that the burden of proof is on the polling firms, not on the public. We seem to be entering a dangerous cycle, where there is a lot of awful information floating around, and no one has the ability to sort the good from the bad. This is as much a problem with what the public is demanding as what the firms are supplying. The public should beware, and commentators should be very cautious about taking firms' power point slides at face value, until some basic methodological questions are answered transparently.

Mapping Development | WRI's "Funnel the Money"

The World Resources Institute, a global environmental think tank based in Washington DC, is providing maps that allow a visual comparison of data for the countries in the South Caucasus. Called "Funnel the Money", it seeks to chart development within countries, and also track allocation of resources from the central government by providing regional comparisons. The intended users are decision-makers, development specialists and informed citizens, and this interactive tool tries to help them "allocate resources to reduce poverty".

The project remains in draft form, and, we hasten to add, is only as good as the source data. The interface is a little clunky, too.

Still, it is an entertaining way to explore what is going on. You were wondering about the number of Georgian centenarians (those over 100 years old), and how they are distributed across the country?

Here are the male centenarians according to the 2002 census:

And here are the female centenarians. Note that there are many more female centenarians, i.e. that the colors should not be compared to the illustration above.

So apparently there are as many female centenarians in Imereti as there are males in the entire country. And Adjaria has a high proportion of both male and female centenarians.

This is just one of many maps you can extract. What is particularly useful is that even the default shows three maps at the same time, allowing for a good visual comparison.

What would we like to be added? A slightly friendlier interface, an opportunity to compare countries, and maybe a disclaimer regarding data quality. As it is, this still seems to be designed from the data end, rather than from the perspective of the users.

The website notes that they are welcoming comments, a WRI team was just out to visit the Caucasus, so we are hoping for an updated version soon. For now, find the site here.

PISA Test | how are Azerbaijani schools doing?

OECD has just published their 2006 PISA results, which stands for "Program for International Student Assessment". In PISA, 15-year olds are tested for basic abilities in various fields. The 2006 round focused primarily on science learning. A little more than 60 countries participated, including Azerbaijan. Georgia and Armenia did not take part.

Alas, the news is sobering. While basic education still reaches the majority of the population, the quality of that education seems limited.

This is demonstrated, for example, by the proficiency levels on the science scale. About 20% of students in Azerbaijan only reach the first proficiency level. This is better than Argentina, Brazil and Tunisia (all 28% not managing to go beyond the first level), and way ahead of Quatar (48%) or Kyrgyzstan (58%). Arguably, Azerbaijan isn't even so far from Bulgaria (18%), Montenegro (17%), Romania (16%) or even Serbia (12%).

However, once a higher level is reached, Azerbaijani performance tails off. Only 0.4% of Azerbaijani students managed to reach Proficiency Level 3 [out of a total of 5 levels] -- that is a disappointing result, especially for a country that was part of a Soviet tradition of teaching. Even Kyrgyzstan is doing better (0.7%), as is Tunisia (1.0%), Quatar (1.6%), let alone Brazil (3.4%), Argentina (4.1%), Romania (4.2%), Chile (8%), Russian Federation (15%), USA (18%), with an OECD average of 20.3 and then that bunch of European states, including the Netherlands that have more than a quarter of their students reaching this proficiency level.

This is genuinely bad news: essentially science education in Azerbaijan has broken down, and lots needs to be done to even catch up. Note the strong divergence between Azerbaijan and Russia. And the same is true for reading ability: only 3.4% of Azerbaijani 15 year-olds reach the 3rd Proficiency Level (where the OECD average is about 28%).

While some of these results may be due to a lack of experience with testing, or even poor translation, the findings suggest where oil revenues could be invested to great use. On that level, it's commendable that Azerbaijan actually took part in PISA -- a very courageous step that yields concrete policy recommendations.

This is no more than a cursory analysis. The datasets are comprehensive and allow gender comparisons, as well as a review of various other indicators.

Exit Polls | a good idea?

With upcoming elections in Georgia, the attention is back on a theme that otherwise often gets neglected: what does the Georgian electorate want?

One of the ideas is to conduct an exit poll, to track the scale of any potential manipulation. You ask a representative sample coming out of the polling station who they voted for, and that should give you a good idea about the electoral results.

According to several people, the Georgian government very much would like such an exit poll. One reason, it is said, is that at least one opposition candidate is considering financing his own exit poll, and getting a large exit polls supported by international donors may counterbalance any biased results that have been paid for by a single candidate.

At face value, this seems like an attractive idea and it has a number of supporters. After all, all you're doing is triangulating, helping to verify what actually happened on E-Day and more information always seems better than less.

But where the trust in the election administration is limited, the risks of exit polls far outweigh any potential benefit. As the head of one organization working in the elections field pointed out, it would "be like fighting fire with fire".

Voters exiting a polling station may not actually tell the truth of who they voted for. This can have various reasons: the social acceptability of their choice, fear for jobs, the first impressdon that the interviewer makes, plain intimidation.

Therefore exit polls can easily be off by 5%, or more. Now imagine one candidate wins the first round legitimately with 52%, but the exit polls only show 47% support, because of such bias, or skewed sampling. The opposition will believe that the election has been stolen, although results simply were within the margin of error.

Ultimately there is no substitute for a regular, disciplined conduct of elections, with citizens actively participating to guard their own vote. If polls were an alternative, there would be no need for the entire elections rigmarole.