Index of Economic Freedom: a critical time lag?

Yesterday the Heritage Foundation released the Index of Economic Freedom 2009. The Index is published annually by the Heritage Foundation together with the Wall Street Journal for over 150 countries of the world (183 in 2009).

But the Index may be misleading, as the analysis of one of the index components reveals that some data that goes into it seem to be outdated. The index is calculated as an average of 10 indicators that measure the level of the economic freedom of a country. The indicators include: business, trade, fiscal, monetary, investment, financial and labor freedoms, as well as government size, property rights, and freedom from corruption.

To measure the last indicator, the Heritage Foundation uses the Corruption Perception Index produced annually by Transparency International. Data sources for CPI are surveys of business people and expert assessments. The authors of the Index of Economic Freedom take CPI and multiply it by 10, since the other 9 indicators that make up the Index are measured on a scale from 0 to 100.

One would expect that the Index published early in the year is based on the data from the previous year. We can expect that the 2009 index reflects the 2008 situation, even though calling this “a 2009 index” is misleading. However, the data goes back even further than 2008.

Compare the Index of Economic Freedom’s Freedom from Corruption score and CPI, which is used as the basis for the former, for Azerbaijan:
The CPI 2007 methodology explains that CPI combines assessments from the last two years – 2007 and 2006. This means that the Freedom from Corruption indicator in the 2009 Index of Economic Freedom is based on the figures of 2007 and 2006.

The authors of the Index use June 30 as the cut-off date for the inclusion of any information in the next year’s report. This means that all information that became available after June 30, 2008 was not included in the 2009 Index. That’s why the most recent CPI data, released by Transparency International in September 2008, did not go into the 2009 Index.

The Index of Economic Freedom is widely accepted as an annual assessment of a country’s performance. As such, it stirs discussion within society, and usually country’s previous scores are used for comparison. The FAQ section of the homepage for the Index states that “some factors are based on historical information." But what is the chance that an Azeri journalist or an ordinary citizen, who learns about the country’s Index for a given year, read the FAQ?

Also, one may ask how consistent and representative a mix of historical and current information (some Index components are based on the recent figures) is?

The credibility of international assessments like this and their impact on the governments, media and civil society across the world would only increase if they are consistent and based on an updated analysis.

Bertelsmann Transformation Index | Using a New Interactive Tool to Analyze the Caucasus

Many of our readers know of both our quibbles with indexes, but also our steadfastness when it comes to posting about them. The Bertelsmann Foundation released its trademark index, the Bertelsmann Transformation Index (BTI) (PDF), which according to its producers, is "the global ranking of the quality of democracy, the market economy and political leadership in 125 developing and transformation countries."

The BTI itself is a the example of what political scientists would call non-parsimonious. The BTI is a combination of two sub-indexes, which are in themselves made up of a wide number of indicators. These indicators in turn are based on an even greater number of variables. I don't want to bore you, so I won't go into details but refer you to the methodological report (which does a great job of documenting their approach). To boot, each of the 125 countries has a 25 page report.

However, despite its lack of parsimony, the BTI provides several good ways of visualizing the developments in the Caucasus -- though, as always, indices have their limits.

Scales run from 0 - 10, with 10 denoting the highest score. Georgia ranks 6.60 (38th) on the Status Index and 6.36 on the Management index (23rd). It gets strong upward ratings in the trends as well. Armenia follows behind Georgia with 6.14 (41stt) and 5.41 (56th). However, it shows no changes, in terms of trends. Azerbaijan lags behind with a score of 4.51 (87th) and 3.83 (99th) and also shows no significant change in terms of the trends.

The next three graphs show the seventeen main indicators that make up the ranking divided into the "Status" and "Management" parts broken down by country.






One way of simplifying and comparing the information in the BTI is a unique interactive tool, called the transformation atlas, which can be downloaded from the Bertelsmann website. While at first slightly difficult to interpret, the atlas provides a good visualization of individual countries, and tells a convincing story about many of the countries development problems. The averages, however, are less useful.



This image shows Georgia compared to Romania (Georgia in blue). The images shows neatly how uneven Georgia's development has been in comparison with a recent EU accession state like Romania. The places where the area in blue is small denotes a lower score. Georgia diverges significantly from Romania here on the key issues of rule of law, socioeconomic development, welfare regime, political and social integration and sustainability of transformation, among others. Since the sustainability variable pertains to the quality of education and research institutions in the country and environmental protection, we agree that Georgia needs more work on these areas. All of these areas sync well with what commentators have noted to be many of Georgia's shortcomings. Georgian does notably well on resource efficiency and steering capability, which address the ability to implement change -- something Georgians have certainly been doing.

The Armenia comparison shows the change over time function. What stands out in the Armenian case, as compared to the Georgian or even the Azerbaijani is the lack of change over time, particularly since 2003, where much greater change is noticeable in other countries in the region. Of interest here is that there has been no change in the socioeconomic ranking despite Armenia's status as the "Caucasian Tiger." However, the socioeconomic development indicator is more concerned with income gaps across the population and social exclusion, and all of the South Caucasus countries do poorly -- scoring 4 across the board.


A comparison of Azerbaijan and Kazakhstan yield interesting findings related to oil rich countries. The first obvious fact is that Azerbaijan looks theoretically very similar to Kazakhstan. However, Azerbaijan finds itself at a slightly lower level than Kazakhstan. Unfortunately, multiple comparisons cannot be carried out at the same time, so we cannot examine this longitudinally with multiple countries. Unsurprisingly, Azerbaijan scores high on stateness and economic performance and other economy related variables and extremely low on indicators related to democracy (upper left corner). Surprisingly Azerbaijan has higher than expected scores (4) on the rule of law.

We encourage our readers to explore more on their own!