CRRC’s Fifth Annual Methodological Conference: In Search of Methodological Innovation

CRRC’s fifth annual Methodological Conference took place on June 23 and 24, 2017 in Tbilisi. This year the conference’s focus was on policy analysis in the South Caucasus, and the search for methodological innovation. Over 50 participants representing institutions in the United States, United Kingdom, Georgia, Azerbaijan, Armenia, Russia, and Canada attended.

Alexis Diamond of the Keck Graduate Institute (KGI), San Francisco gave an opening address for the conference titled: Deliberate ignorance: The dangers of knowing too much too soon. The talk covered a wide range of issues in evaluation, however, emphasis was placed on the importance of honest evaluation.

The first day of the conference had four sessions, with papers on a wide variety of subjects from the geographies of polarization and inequality in Tbilisi to a field experiment on marshrutka safety and a machine learning approach to profiling tax awareness in Armenia. 

The opening slide of David Sichinava’s presentation on Spatial Patterns of Emerging Inequalities in Tbilisi, Georgia.

The second day of the conference was dedicated to workshops. Alexey Levinson of the Levada Center, Moscow lead a workshop on open-ended group discussions and Aaron Erlich of McGill University discussed the fundamentals of multiple imputation. The conference also included workshops on case studies in public health, web surveys, and synthetic controls.

Aaron Erlich discussing why and when to use multiple imputation.

For more information, the full conference program can be accessed here.

Back to the USSR? How poverty makes people nostalgic for the Soviet Union

A recent CRRC/NDI survey asked whether the dissolution of the Soviet Union was a good or bad thing for Georgia. People’s responses were split almost evenly: 48% reported that the dissolution was a good thing, whereas 42% said it was a bad thing for the country. Such a close split raised questions in the media about why people took one view or another.

While it is tempting to explain assessments of a past event, such as the dissolution of the Soviet Union, using people’s attitudes towards foreign policy issues, this blog post only looks at respondents’ socio-demographic and economic characteristics and some reported behaviors that could potentially shape their attitudes. Specifically, we look at the impact of gender, age, education, ability to speak English and Russian, frequency of internet use, settlement type and the number of durable goods a household possesses, out of the ten durables the survey asked about: a refrigerator, color TV, smartphone, tablet computer, car, air conditioner, automatic washing machine, personal computer, hot water, and central heating. We interpret the number of durables owned as a measure of the households’ economic status. Surely, this measure is not perfect and gives us only partial information about the household’s economic conditions. However, this is the best available measure from this particular survey, provided that many people do not like reporting their income or expenditures, or do not provide accurate information on these.

The chart below shows the results of a logistic regression model which predicts the odds of a respondent saying that the dissolution of the Soviet Union was a good thing for Georgia. The dots on the chart indicate point estimates for each independent variable, and the lines show 95% confidence intervals. If a line does not cross the vertical red line, we are 95% confident that the variable has an impact on the dependent variable, i.e. the belief that the dissolution of the Soviet Union was a good thing for Georgia. The further a horizontal line from the vertical red line, the larger the effect of the variable.

The model shows that gender and the ability to speak either English or Russian do not influence people’s assessments of the dissolution of the Soviet Union. As one might expect, age has a significant, negative impact: the older a person is, the lower is the probability that s/he will express a positive attitude towards the dissolution of the USSR. Education, frequency of internet use and possession of durables have the opposite impact: people with tertiary education are more likely to assess the dissolution positively than people with less than tertiary education. Likewise, people, who use the internet at least once a week assess the dissolution more positively than people who use the internet less often or never. Also, the more durables a household owns, the higher the probability of assessing the dissolution of the Soviet Union as a positive event for Georgia.

As expected, settlement type also matters: the chart shows the effect of living in Tbilisi, other large towns, predominantly Georgian-speaking rural settlements and ethnic minority settlements, which are compared to small towns – the reference category. Large towns include six cities with more than 40 thousand people, whereas smaller urban settlements are grouped into a small town category. We define an ethnic minority settlement as a location in which 40% or more of the inhabitants are ethnic minorities. Normally, these are towns and villages with large Armenian or Azerbaijani populations in the Kvemo Kartli, Samtskhe-Javakheti and Kakheti regions.

While residents of large towns and rural settlements have similar opinions about the dissolution of the Soviet Union as residents of small towns, Tbilisi residents are more likely to assess the dissolution positively. In contrast, those living in minority settlements tend to assess the dissolution negatively.
Based on the above, we conclude that age, education, frequency of internet use, possession of durables and settlement type influence an individual’s assessment of the dissolution of the Soviet Union. As a next step, we tested whether age, education, frequency of internet use, and possession of durables influence individuals’ attitudes differently in different settlement types.

The analysis shows that the impact of age, education and internet usage does not vary by settlement type. However, we observe a very different picture in the case of household possessions: possessing more durables increases the probability of positive assessment of the dissolution of the USSR in all settlement types except for (non-minority) villages and small towns. Its impact is largest, however, for residents of ethnic minority settlements. If an individual living in such a settlement has no durables, his or her probability of assessing the dissolution of the Soviet Union positively is below 20%. However, as the number of durables in the household increases, the probability of a positive assessment increases nearly linearly, and exceeds 60% when the household owns all ten items asked about on the survey.

Hence, we conclude that age, education, settlement type, and economic conditions significantly influence people’s assessments of the dissolution of the Soviet Union. The impact of a household’s economic situation is largest in ethnic minority settlements. Therefore, economic deprivation, arguably caused by and interrelated with a number of other factors, seems to be the most important driver of negative assessments of the dissolution, rather than minority status per se.  

To have a closer look at the CRRC/NDI data, visit CRRC’s Online Data Analysis tool.

Most households in Georgia report limiting food consumption, despite economic growth

According to the World Bank, GDP in Georgia increased from USD 10.1 billion to USD 13.9 billion between 2009 and 2015. Despite this growth, according to CRRC’s Caucasus Barometer survey (CB), the share of those who reported not having enough money to buy food on at least a weekly basis did not decrease between 2011 and 2015. This blog post shows how this finding differs by settlement type and reported household income.

As the chart below shows, the general picture did not change between 2011 and 2015. Only about one third of the population claims it never happened during the 12 month prior to the survey that they did not have enough money to buy food they or their family needed. More than one third report encountering such difficulties periodically and about a quarter monthly or more often.

Note: Answer options “Every day” and “Every week” have been combined for the charts in this blog post.

Taking into account the margin of error, the share of people who reported not having enough money to buy food every week or more often is approximately the same in different settlement types. Importantly, the most common response in Tbilisi and the second most common response in other urban settlements is “Never”. This answer is, however, reported by less than half of the population of these settlement types.

Note: Answer options “Don’t know” and “Refuse to answer” were excluded from the analysis.

                       
A lack of money for food logically suggests a low income. The chart below shows that the higher the reported household income, the higher the share of the population reporting never being in a situation when they did not have enough money for food. According to CB 2015, 61% of the population reported their household income was less than USD 250 the month prior to the survey.

CB data show that a large share of households in Georgia have financial difficulties supporting their families’ primary needs and a majority struggle with not having enough money for food at least some of the time.

To have a closer look at the Caucasus Barometer data, visit CRRC’s Online Data Analysis tool.

Georgia: Disapproval Rising for NATO Membership

[Note: This piece was originally published on Eurasianet. Dustin Gilbreath is a policy analyst at CRRC-Georgia. Rati Shubladze is a researcher at CRRC-Georgia. The views expressed in this article do not necessarily reflect the views of either CRRC-Georgia or the National Democratic Institute.]

NATO recently recognized Georgia’s contributions to peacekeeping missions from Afghanistan to Kosovo by holding a session of the alliance’s parliamentary assembly in Tbilisi in late May. The occasion reinforced the hopes of Georgian leaders that their country can one day soon gain admission to NATO. However, polling in the lead up to NATO’s parliamentary assembly also sheds light on a trend that could potentially hinder its membership bid.

Public support in Georgia for the country’s NATO membership bid remains strong. A recent survey CRRC-Georgia carried out for the National Democratic Institute shows that 68 percent of Georgians support the government’s goal of joining the alliance. If Georgia were a NATO member, this would be the third highest level of support of any member state polled in a recent Pew Research Center survey.

Yet, CRRC and NDI’s data also shows that disapproval with the prospect of membership is rising. Back in 2012, roughly a quarter of the public was uncertain over whether the country should join NATO; since 2015, however, only about one in 10 have reported uncertainty. Over the same period, disapproval of NATO membership doubled from about one in 10 Georgians to roughly one in five.

This trend has at least two potential explanations.

First, people who used to report that they are not sure about membership might have always been opposed to the Alliance. Rather than telling interviewers this, they felt social pressure not to say so, because they perceived NATO support to be popular in Georgia. This phenomenon, being shy about reporting unpopular opinions to survey interviewers, is common, and is known as social desirability bias.

If this explanation is correct, then the shift from uncertainty in response to disapproval is a sign of a trend in Georgian society and its foreign policy discourse: anti-Euro Atlantic views are more widely accepted or at least perceived to be more socially acceptable than in years past. Over the past couple of years, politicians have expressed less confrontational views towards Russia, at least when compared with the virulently anti-Russian rhetoric of former president Mikheil Saakashvili and his United National Movement, which lost its parliamentary majority in 2012. This change in discourse might contribute to the trend, making it more widely acceptable to express views that are not pro-Western. While beyond the scope of this article, Russian propaganda too could be playing some role.

A second possible explanation is that a significant number of those who were previously undecided are now making up their minds: no longer sitting on the fence, they have decided that the actual or potential costs of NATO membership are too great, or the chance of NATO membership too low, to make the required sacrifices.

Georgia is a small country and, even in absolute terms, it contributed more soldiers to NATO’s mission in Afghanistan than any other non-member state. This contribution has not come without a cost: over 30 Georgian soldiers have died in Afghanistan and hundreds have been wounded.

The potential for NATO membership to incite Russia’s ire weighs heavily in the minds of those who disapprove. When those who reported opposing Georgia’s NATO bid were asked why they disapprove, the most common response was that it will cause conflict with Russia.

Despite Georgia’s sacrifices, membership in the Alliance seems distant to a majority of Georgians. Since the 2008 Bucharest Summit Declaration, which stated that Georgia and Ukraine would someday become members of the Alliance, a Membership Action Plan – a first step towards membership – has proven elusive. This is reflected in public opinion about when Georgia will join NATO: 16 percent think the country will never join, and an additional 38 percent are uncertain if or when the country will be offered membership. A majority of those who are uncertain about a membership date favor Georgia’s NATO bid.

Even though disapproval of Georgia’s NATO bid may be rising, the head of the NATO Parliamentary Assembly has suggested that Georgia is more prepared for membership than even some member states. Notably, Georgian military expenditure has consistently exceeded 2 percent of GDP, the level required of NATO members, despite the fact that only five member states meet this spending target. On top of this, a full 80 percent of those polled think that military spending should either stay the same or increase.

If the Alliance is dedicated to its 2008 Bucharest commitments, it should make its intentions clear to the Georgian public. The lack of a clear signal from the Alliance seems likely to only keep stoking uncertainty and disapproval of members among the Georgian public.

Awareness of EU aid and support for EU membership in Georgia

The EU provides a wide variety of aid to Georgia. Within the European Neighbourhood and Partnership Instrument (ENPI) alone, EUR 452.1 million was allocated for Georgia in 2007-2013. What, if any, role does this aid play in influencing the population’s attitudes towards the European Union? This blog post looks at the awareness of the population of Georgia about the EU’s aid to the country, on the one hand, and support for Georgia’s membership in the EU, on the other hand, using the 2015 CRRC/EF survey on Knowledge of and Attitudes towards the EU in Georgia (EU survey). The findings suggest that those who are more aware of EU aid are also more likely to support Georgia’s membership in the EU.

About 1/10th (13%) of the population of Georgia think the EU does not assist Georgia in any way, while about a third (31%) could not or would not answer the question, “In your opinion, what are the main types of aid the EU currently provides to Georgia?” Of the 56% who named a specific type of aid, humanitarian aid and investment in Georgia’s economy were the most frequent answers.

Note: A show card was used. The answers do not add up to 100% since respondents could choose up to three answer options. 

This question was recoded for further analysis. In a new variable, “Does the EU provide any aid to Georgia?”, all answers where a certain type of aid was mentioned were combined into the category “The EU provides aid,” while options “[The EU] does not provide any aid”, “Don’t know” and “Refuse to answer” were combined into the category “Don’t know/Does not provide aid.”

As the 2015 EU Survey report and CRRC’s previous blog posts have highlighted, in 2015, 61% of the population of Georgia reported that they would vote for the country’s membership in the EU, if a referendum was held tomorrow. Seventy-five percent of those saying that the European Union provides any type of aid to Georgia report that they would vote for EU membership. Only 44% of those saying the EU does not provide aid or answered “Don’t know” report the same.

The 56% of the population who named a specific type of aid that, in their opinion, Georgia receives from the EU, were then asked which groups benefit most from this aid. Politicians and high level officials in Georgia were named most often as the group that benefits most from EU aid to the country.

Note: The question was asked only to those who reported that the EU provides aid to Georgia. A show card was used for this question. The answers do not add up to 100% since respondents could choose up to two answer options. 

These findings suggest that awareness about EU aid to Georgia matters when it comes to people’s support for Georgia’s membership in the EU. Those who believe that the EU provides specific types of aid to Georgia tend to report more often that they would vote for Georgia’s membership in the EU.

To learn more about the population’s support for Georgia’s membership in the EU, visit CRRC’s Online Data Analysis tool, or take a look at some of CRRC-Georgia’s blog posts on the subject, here.

Who is more intelligent and sincere? CRRC interviewers’ assessments

It has been only about two decades since pollsters started using paradata, including interviewer assessments of the conducted interviews and respondents’ behavior and attitudes. Such assessments are collected in the process of most of CRRC-Georgia’s surveys. Immediately after each interview, interviewers are asked to assess various aspects of the interview, including respondents’ sincerity and intelligence. Using CRRC’s 2015 Caucasus Barometer (CB) data for Armenia and Georgia, this blog post explores how such assessments differ depending on respondents’ demographic characteristics and their households’ economic situations.

It is important to highlight that these assessments can hardly provide reliable information about the actual intelligence or sincerity of respondents, and there is no doubt that interviewers’ perceptions of intelligence are very subjective, thus they cannot be attributed to any objective characteristics. Still, when analyzed properly, they can tell us a lot about a wide range of issues, from the process of interviews to the dominant stereotypes and biases in a given society.

In the CB 2015 assessment forms, interviewers were asked to rate respondents’ intelligence. In both Armenia and Georgia, interviewers tend to rate the intelligence of people who live in the capital and other urban settlements higher than the intelligence of those who live in villages.

 Note: The original 5-point scale was recoded for the charts in this blog post, with answer options “Intelligent” and “Very intelligent” combined into the category “Intelligent” and answer options “Not at all intelligent” and “Not very intelligent” combined into the category “Not intelligent”. Responses “Don’t know” and “Refuse to answer” were excluded from the analysis in the charts in this blog post.

Although, overall the respondents are reported to be rather sincere during the interviews, interviewers report the answers of people living in the capital and other urban settlements to be slightly more sincere compared to the answers of people living in villages.

Note: The original 11-point scale was recoded for the charts in this blog post, with codes 0 through 3 combined into the category “Not sincere”, codes 4 through 6 combined into “Average” and codes 7 through 10 combined into the category “Sincere”. 

There are even more pronounced differences by respondents’ level of education. Interviewers rate people with a lower level of formal education both as less intelligent and sincere. In both countries, people who report having secondary technical education or secondary or lower education are twice as likely to be placed in the category “Average” than those with a higher than secondary education.

 Note: Answer options “No primary education”, “Primary education”, “Incomplete secondary education” and “Secondary education” were grouped into the category “Secondary or lower”; “Incomplete higher education”, “Completed higher education” and “Post-graduate degree” were grouped into the category “Higher than secondary”. 

In contrast to the assessments of intelligence, interviewers’ assessments of respondents’ sincerity vary less by the respondents’ level of education. Both these assessments, however, are very similar in Armenia and Georgia.

Interviewers’ assessments of respondents’ intelligence and sincerity also differ according to the reported economic situation of the respondent’s household. Interviewers tend to perceive those who are better-off economically as more intelligent. For example, interviewers report people living in households that can afford to buy expensive durables to be intelligent almost three times more often than people who do not have enough money for food, in both Armenia and Georgia.

Note: Answer options “We can afford to buy some expensive durables [like a refrigerator or washing machine]” and “We can afford to buy anything we need” were grouped into the category “Enough money for expensive durables”.

Those who report being in the best economic situation tend to be rated as sincere more often than people in all other groups.

Further research is needed to find out to what extent interviewer assessments reflect the actual situation, i.e. whether those who are better off, or urban dwellers, are actually more intelligent or sincere, and vice versa. The findings presented in this blog post show that interviewers have rather strong opinions about what groups of respondents are intelligent and sincere, and the assessments are very similar in Armenia and Georgia. Interestingly, there is a high positive correlation between the assessments of respondents’ intelligence and sincerity in both countries, suggesting another interesting topic for further research, namely, to what extent intelligence and sincerity are seen by interviewers as interrelated characteristics.

To find out more about CRRC’s Caucasus Barometer and other surveys, check out our online data analysis tool.

Air Pollution in Tbilisi: What the Data Says

[Note: This is a guest blog post by Dr. Hans Gutbrod, the former Director of CRRC.]

With the recent debate on traffic safety, it may also be a good time to highlight the issue of air pollution in Tbilisi – especially as municipal elections are coming up, and citizens (and candidates) may ask themselves what issues the election should focus on.

The data is clear: citizens care about pollution and the environment. In the June 2016 survey that CRRC conducted for NDI-Georgia, pollution was seen as a key concern for residents of Tbilisi. In the capital, 38% named it as one of the most important infrastructural issues, placing it ahead of all other issues related to infrastructure. In general, all generations care, but the young a bit more. In Tbilisi, about 40% of those under 36 years of age mentioned pollution as an issue compared to 32% of those 56 and older.

Politically, the topic also seems to resonate, though less strongly with voters who tend towards the big parties. Only 33% of Georgian Dream voters and 35% of United National Movement voters reported that pollution was the number one infrastructural issue in Tbilisi (Note: the party landscape has changed since the survey). This may in part be a result of the small sample size once you do crosstabs, but could also suggest that the environment and pollution may be an issue to mobilize and rally voters around. Supporting this contention is the fact that 40% of undecided voters in Tbilisi named the issue as the most important one in Tbilisi.

Yet how bad is the situation really? It is not so easy to find out. The government does collect data at three measuring stations that have been donated by Japan, but it is made available one day late, in PDFs, and even those who have interpreted lots of data will need a significant amount of time to decipher what is going on.

A very quick glance on some random days in February suggests that the pollution in Tbilisi repeatedly exceeds limits that are considered healthy – often by a multiple.

Whatever the policy prescription, one sensible next step for citizens and parties to demand, and for the government to take, is to make this data accessible straightaway, live. Having the data, citizens could decide on what to expose themselves to, when. To have a good debate on policy, we need good data. Thus, a sensible suggestion to politicians whenever and wherever you meet them is to request that public pollution data be made available in real time.

Debts and Loans in Georgia (Part 2)

The first part of this blog post showed that people who report being in a worse economic situation are more likely to have debts in Georgia. In the second part of this blog post, a new variable is added to the analysis, “Does anyone owe you any money?”

While 46% of the population of Georgia report having debt, only 20% report that someone owes them money. In the latter, group, there are no differences by gender and settlement type, but there are differences by age. People between 36 and 55 years of age are more likely to say that someone owes them money. As seen in the first part of this blog post, people in this age group are also most likely to report they have personal debts.

The cross tabulation of the questions about having debt and being owed money shows that people who are owed money are slightly more likely to have debts.

A new variable, “Debts and Loans,” was created to group people into four categories based on the two CB questions discussed above.

Forty four percent of the population of Georgia are part of the largest group who report neither having debts nor being owed any money. These people are neither better off nor worse off compared to the population on average. The second largest group has debts but no one owes them money. They appear to be in the worst economic situation, with the greatest share of people saying they do not have enough money for food and for clothes in comparison to other groups. The two smallest groups are people who say someone owes them money. The two groups who have no debts appear to be in a relatively good economic situation, with the largest shares of people saying they can afford expensive durables.

Based on the findings presented in both parts of this blog post, debts are approximately twice as common in Georgia as being owed money. Yet, the largest share of the population of the country are those who report neither having debts, nor being owed money.

To look at these issues in more detail, explore the Caucasus Barometer data at CRRC’s Online Data Analysis platform.

Debts and Loans in Georgia (Part 1)

In Georgia, where, according to the World Bank, a third of the population live on under USD 2.5 per day, poverty and unemployment are consistently considered the most important issues facing the country. For those who are struggling financially, borrowing is a widespread coping mechanism. While access to credit can have benefits, debt can also have psychological costs, such as increased stress and anxiety. CRRC’s 2015 Caucasus Barometer (CB) data show interesting patterns about having personal debts in Georgia. The first part of this blog post focuses on the characteristics of those who report having personal debts in Georgia, while the second part looks at the money lending patterns, as well as reported well-being of people who are owed money or who borrow it.

In response to the question, “Do you currently have any personal debts?” which asks about all types of debt a person may have, 46% of the population report having debts and 53% say they do not have any. There are no large differences by settlement type. People between 36 and 55 years of age report having debts more frequently than people in other age groups. Men report they have debts slightly more often than women.

Note: The charts in this blog post do not include answer options “Don’t know” and “Refuse to answer,” which constituted 1% of responses.

People reporting a more difficult economic situation in their household are more likely to say they have debts. While 55% of people who state they do not have enough money for food report having debts, 28% of people who have enough money for durables report the same.

Note: Answer options “We can afford to buy some expensive durables like a refrigerator or washing machine” and “We can afford to buy anything we need” were combined into the category “Can afford to buy expensive durables” on the chart above.

In the second part of this blog post, which will be published on Monday, patterns of both borrowing and lending money will be discussed.

To have a closer look at the Caucasus Barometer data, visit CRRC’s Online Data Analysis platform.

Rising expectations: People report more positive expectations about MPs immediately after elections

Research suggests that voters not only become more knowledgeable about political issues, but also more politically engaged during electoral campaigns. CRRC/NDI survey data also suggest that during the periods immediately following elections, there are more positive expectations about elected officials. These, however, do not last long after elections.

A citizen’s knowledge of which Member of Parliament (MP) represents her or him ebbs and flows with election cycles in Georgia. In March 2016, three and a half years after the most recent parliamentary elections in October 2012, only 31% of the population of Georgia answered correctly who their majoritarian member of parliament was at the time. A month after the October 2016 parliamentary elections, in November 2016, the respective share nearly doubled (57%). The findings before and after the 2012 parliamentary elections are similar. In the period between the elections, knowledge of which MP represented a constituent declined.

People’s expectations of their MPs also oscillate with the electoral cycle, with higher expectations immediately after elections. After both the October 2012 and October 2016 elections, the share of those reporting that MPs will serve people’s interests increased almost two-fold compared to early spring of the election year. Expectations that MPs will serve only their own interests have a tendency to decrease immediately after elections. However, they gradually increase later on. The expectations that MPs will do what their political party will tell them to also decrease immediately after the elections, although the gaps are smaller.

Note: In the survey waves from February 2012 through April 2014, the question was asked about majoritarian members of parliament. Since April 2015, the question was asked about members of parliament in general.

Positive expectations also increased after the October 2016 elections in respect to whether the newly elected MPs will take into account the opinions of regular people – “people like you,” as it was worded in the questionnaire. On the November 2016 survey, 63% either completely or somewhat agreed with this opinion, while the respective share was only 28% in March 2016, when the same question was asked about the MPs that were in office at the time.

This evidence suggests that people in Georgia become more optimistic about members of parliament in the months immediately following elections, believing that elected politicians will serve people’s interests. However, as time passes, they become disillusioned and their expectations become more skeptical.

To explore the CRRC/NDI survey findings, visit CRRC’s Online Data Analysis portal.

How many Tetri are in a Lari? The importance of municipal statistics for good governance

[Note: This post was co-published with Eurasianet and authored by Koba Turmanidze, CRRC-Georgia's Director.] 

The government of Georgia committed itself to collect and publish policy-relevant data in a timely manner under the Open Government Partnership. Yet while most ministries and state agencies are happy to provide national-level statistics, they often fail to break them down to the municipal level. 

As a result, if you think about it in monetary terms, the current system means that officials do not know how many tetri are in a lari. 

Reliable municipal statistics can contribute to good governance in several important ways. First, municipal-level data can let citizens assess the quality of state services they receive compared to other municipalities, or to the national average. Second, municipal-level data can help policy makers improve the targeting of programs, and therefore, spend public money more efficiently. Third, it can help both government and citizens evaluate the successes and failures of municipal governance.

The following scenarios highlight the benefits of comprehensive data on a local level: 

Imagine you want to move from one town to another for a better job. You consider moving with your spouse and a child of school age. However, your spouse does not want to move, arguing that your town has much better public schools than the new one. In such cases, it would help if you could look at municipal-level education data to see if schools in the two towns are similar or different in terms of the student-teacher ratio, exam scores, the success rate on national examinations, etc. 

Imagine you are a civil servant and are working with an investor to build a new factory in a municipality. You want the factory to be built in the municipality where its social impact will be highest. To persuade the investor, you use municipal statistics to demonstrate that the municipality of your choice has high unemployment, yet its labor force is younger and better trained than in comparable municipalities. 

Imagine you are an analyst in a think tank and your task is to advise the government on whether to extend a poverty reduction program or not. The government claims the program helped to reduce poverty by two percentage points nationwide, but you have reasons to suspect that the reduction happened in certain settlements, whereas in others the program had a negligible impact. You look at relevant data on the program and poverty statistics, and conclude that the program’s effect across municipalities was truly unequal. Importantly, it made no difference in the most economically deprived communities. Therefore, you advise the government to redesign the program to improve its impact on the communities with the highest poverty rates, before pouring more money into it. 

Unfortunately, we can only imagine the above. These three scenarios remain purely hypothetical, since, in Georgia, reliable municipal data is rarely available on education, employment and poverty. 

CRRC-Georgia’s repeated interactions with a multitude of state agencies over the past three years have uncovered at least three problems regarding municipal data. First, when municipal data of potentially good quality exists, it is often not processed and made available to the public. Second, if municipal data is accessible, its quality is often questionable. Third, municipal data often does not exist at all, since the responsible agency does not recognize its value, or is unable to collect it due to lack of relevant training. 

Below are three concrete cases that correspond to these three problems: 

Case 1: Until recently, the National Assessment and Examination Center (NAEC) maintained detailed and high-quality data on the results of the Unified National Exams (UNE) for many years. The data allowed one to see which municipality’s and even school’s students were most successful in the exams. Such data was not proactively shared on the Center’s website, but was available upon request. The situation changed when the Center introduced electronic applicant registration in 2011. Under the new system, an applicant’s place of residence and school was no longer recorded. However, the Center could still identify the municipality based on an applicant’s ID number. 

For the 2015 UNE results, NAEC processed the data this way and made the data file available on its website. However, when CRRC-Georgia requested the same data for 2016, the Center turned the request down, arguing it no longer processed data per municipality, and would not do so for our sake. As a result, it is no longer possible to analyze municipal- or school-level performance and map it as we did in this blog post. At the time, this map drew attention to large differences in educational attainment countrywide, including an important fact – that Unified National Exam scores in Upper Ajara were among the worst in the country. In part in response to this fact, AGL, a Norwegian company building a hydro-electric dam in upper Ajara created a tutoring program for students in the region to help them prepare for the exams. If the NAEC withholds such data, it will hinder the ability of interested parties to spot trends and develop remedial policies.

Case 2: The Social Service Agency (SSA) is a leading organization in the country in terms of providing access to comprehensive data on poverty and targeted social assistance at the national and municipal levels. Among other statistics, the Agency reports monthly data on applicants and recipients of social aid. The 2014 census data, however, casts some doubt on whether the SSA poverty statistics are trustworthy. 

The scale of mismatch between the agency’s data and the 2014 census results is evident from the chart below. Using Geostat’s population estimates, the agency calculates the share of the population registered for targeted social assistance in each municipality. The census was conducted in November 2014, so it is possible to re-calculate the share of those registered for targeted social assistance based on the census data and compare it with the agency’s estimates. 

Let’s take the extreme case of Lentekhi. The agency reported that 44 percent of the population applied for targeted social assistance. When the census data is used, the finding is that 89 percent of Lentekhi residents registered for social assistance. Thus, the SSA estimate was 45 percentage points lower than the actual percentage. The chart below plots the differences between the shares of the population registered for targeted social assistance, as calculated and reported by the SSA on the one hand, and updated calculations based on the 2014 census data on the other, for each municipality in November 2014. Overall, the agency underestimated the share of applicants by 14 percentage points, on average. However, publicly available data has yet to be adjusted based on the 2014 census.    

Case 3: In a number of cases municipal statistics do not exist. Measuring the scale of economic activity (level of employment by employment sector, total value added, etc.) in every municipality would require large-scale surveys that are both time-consuming and expensive. Geostat’s periodic Integrated Household Survey cannot provide this information, due to the cost that such a large sample size would entail. 

However, the Revenue Service (RS) under the Ministry of Finance of Georgia could help solve this challenge. Based on taxpayers’ IDs, the agency can provide information about the number of taxpayers, be it individuals or organizations, and amount of taxes collected in each municipality. This information would serve as a good proxy of economic activity by municipality. However, as the RS told CRRC-Georgia, it can only break the data down for regions. Officials claimed that breaking the data down further was not possible. 

Undoubtedly, the collection and analysis of municipal data requires additional resources. However, the three concrete cases highlighted above show that a little increase in awareness regarding municipal data could go a long way toward promoting better municipal governance in important ways. Investors could have clearer insight into the best investment destinations, for one. Civil society groups would also have better ways to assess the successes and failures of government actions. Citizens likewise would have a better idea of where their place of residence stands compared to other parts of the country, or the national average. Government could have better tools to ensure equal access to services in the country, or to achieve efficiencies in the provision of services. 

All of this is possible. Unfortunately, it is not reality. And as a result, the government doesn’t know how many tetri are in a lari.

Why the civil service? The civil service as seen by civil servants in Georgia

The civil service plays an important role in the development of a country. Thus the competence and motivation of civil servants matter. An online survey of civil servants conducted by CRRC-Georgia for NATO-PDP in December 2015 – January 2016 was one of the first attempts in Georgia to study civil servants’ perceptions of and attitudes towards their job. This blog post provides a brief overview of some of the findings, focusing on reported reasons for choosing to work in the civil service, the advantages and problems civil servants see with their jobs, as well as general assessments of civil servants’ motivation to work.

Slightly over a half of civil servants (56%) reported that the main reason they chose their job was an interest in working in the public sector. Forty-five percent hoped to improve their professional skills. The third most frequently named reason was a hope to improve the situation in their field. Importantly, a rather small share of civil servants (18%) reported a “stable job” as a reason for choosing to work in the civil service, and only 2% named an “attractive salary”.

About half of civil servants named the opportunity to contribute to the development of the country as the main advantage of working in the civil service. The opportunity to acquire new professional skills was the second most frequent answer, while the opportunity to make new connections, work one’s way up within the organization, and work with competent colleagues were named rarely.

Civil servants most frequently assessed the motivation to work of those in the civil service as average (50%). Positive assessments, though, are much more frequent than negative ones (36% vs 15%).

Note: To a certain extent, in response to most of the questions in this survey, civil servants tended to pick answers that can be considered socially desirable. Hence, social desirability bias is likely to have affected other responses as well, including assessments of other civil servants’ motivation to work. In order to avoid this bias in the case of this question, a very general wording was used: “How would you assess civil servants’ motivation to work?” This question measures civil servants’ reported perceptions of their colleagues’ motivation. To a certain extent, social desirability bias likely still influenced responses. However, this influence should be less than if the question had been asked about the respondent herself. 

Should the 50% of “average” assessments be considered a problem? Probably. The state invests financial resources in the improvement of civil servants’ professional skills, and the country, overall, is interested in a high productivity of their work. Undoubtedly, motivation encourages productivity. Thus, higher motivation will also lead to a more efficient use of public resources.

While trying to increase civil servants’ motivation in Georgia, it is important to consider the problems they face. The most frequently named problem is a lack of professionally competent civil servants. A fear of losing one’s job and poor infrastructure come in second and third, followed by a fair number of other problems.

Addressing these problems may improve civil servants’ motivation to work. As a result, the country would benefit from higher productivity among civil servants.

Proposed Reform Could Tilt Electoral Field Toward Incumbents

Irakli Kobakhidze, Speaker of Georgian Parliament, recently outlined proposed constitutional changes. Among them is a switch to a fully proportional electoral system, which civil society groups have long argued for. But this possible switch could tamp down on political competition.

This pending reform would produce two major changes besides the nixing of the first-past-the-post component: the first would re-allocate votes for parties that fail to clear the 5-percent electoral threshold (the percentage needed for a party to gain representation in parliament) to the leading party; the second, which is ironic coming from a party that came to power as a coalition, would prohibit parties from forming electoral coalitions. Taken together, some experts have cautioned that the proposed rules-changes could tilt the electoral playing field in favor of the incumbents, and thus would mark a setback for civil society.

In order to test the claim that the proposed reforms could skew democratic development in Georgia, and how the changes might affect future elections, the Caucasus Research Resource Centers calculated how seats would have been distributed among parties in the past three legislative elections.

The CRRC model looks at three variants, including the mixed system that Georgia currently uses, one which distributes seats through a coalition of a party-list vote, and first-past-the-post races. The CRRC model also applies the criteria that the parliament speaker proposed March 21, under which there is only the proportional, party-list vote, along with the re-allocation rule that gives the leading party the votes received by parties that fail to clear the 5-percent hurdle. In addition, the CRRC model shows how MP seats would be distributed on the basis of a proportional-vote system only with a 5 percent threshold, but without the re-allocation provision.

The CRRC model shows that in the 2008 and 2016 elections, during which there was only one strong party in the country, the incumbent party would have ended up with roughly the same number of seats under the current mixed system and the proportional-only system with the re-allocation rule. The incumbents would lose a significant number of seats, yet still retain a safe majority, under the system of proportional voting without re-allocation.

There would seem to be a genuine danger that the proposed rules changes would give the incumbents a decided advantage in a race in which two parties are otherwise evenly matched.

The 2012 election underscores this point. That race, which matched the Georgian Dream against United National Movement (UNM), was close and intense. When the results are run through the CRRC model, the parliament would have almost the same seat allocation under a fully proportional system as under the current mixed system. However, if the votes for parties that didn’t clear the 5-percent hurdle were re-allocated as Speaker Kobakhidze suggested, it would favor the winner, when compared to a proportional system without the re-allocation rule.

While the proportional system Georgian Dream proposes might appear to mark an improvement over the current system in some cases, under circumstances where there is a high level of political fragmentation, such as is currently the case, the changes would provide a significant boost to incumbent authority.

To highlight this phenomenon, a detailed look at the 1995 Georgian parliamentary elections is useful. In the 1995 vote, over 50 parties vied for seats in the legislature and 62 percent of ballots were cast for parties that didn’t clear the 5-percent hurdle, and thus were ineligible for representation in parliament. Had the electoral system in 1995 been the one that the Georgian Dream is currently advocating, rather than 107 seats, the Citizens Union of Georgia would have won 187 seats in what was a 235-member legislature.

The Georgian Dream, which came to office as an electoral coalition, is also proposing a ban on electoral coalitions. This rule in combination with wasted votes being allocated to the winner is likely to further benefit incumbents.

Each party would either have to run independently, or form a new political party to run together in elections. In practice, this rule obstructs electoral consolidation, which often promotes the formation of stable parliaments in a genuinely pluralistic system. If implemented in its current form, it is likely the new rule would increase the number of seats in parliament awarded to the party with the largest share of the vote.

Even without this change, Georgia’s political parties have already been fragmenting in recent years. The Georgian Dream was once a six-party coalition, but now consists of a single large party (the Georgian Dream) and two minor parties (the Conservatives and the Social Democrats). The remaining parties that formed the original coalition are now independent. Meanwhile, the United National Movement (UNM) has broken up into a number of parties, starting with the 2012-2016 parliament, when UNM members split to form Girchi and New Georgia, and more recently, when a majority of the party’s MPs broke off to establish the Movement for Freedom - European Georgia.

Another matter of potential concern is the question of dark money in the electoral system. It’s possible that incumbents, acting through shadowy intermediaries, could potentially provide assistance to small parties in order to draw votes away from potential competitors. This strategy has already been part of political competition in Georgia. When the UNM controlled parliament, for example, the Christian Democrat party was effectively a satellite party. This strategy may have also been at play in some of the splits from the United National Movement. Both Girchi and New Georgia, after splitting off from the UNM, were able to immediately open up a large number of offices throughout the country. It remains unclear where the money came from to afford such a significant expense.

The proposed electoral reforms could face opposition from civil society, and President Giorgi Margvelashvili could veto them. However, the Georgian Dream holds a constitutional majority in Georgian parliament, meaning it could overturn a veto, as MPs did in response to past vetoes.

For Georgia’s democratic political system to become stronger, the consolidation of parties, not further fragmentation, is needed. The proposed changes to the electoral system are unlikely to encourage consolidation, and instead seem to provide lots of incentives to encourage the continued fragmentation of the political landscape. At best, the proposed changes are two steps forward and one step back. At worst, they could squash political competition in Georgia.

To view the data used in this article, click here.

[Note: This post was originally published on Liberali in Georgian and on Eurasianet in English. Dustin Gilbreath is a Policy Analyst at CRRC-Georgia. David Sichinava is a Senior Policy Analyst at CRRC-Georgia.]

Safer transport options for passengers: Recommendations

In the previous blog posts in this series (see here, here, here, and here), we reported the design and results of a randomized control trial on minibus safety in Georgia. In this blog post, we provide recommendations for the Government of Georgia based on the results of the experiment. First, we recommend that the Government:

• Create an anonymous minibus monitoring program

Telling minibus drivers that they are being monitored for safe driving and may be punished for safety-violations may lead to safer, less distracted driving. The results of the experiment suggest such a program would likely be effective. This suggests that the government has an opportunity to implement a small program, which could have an important impact on making minibus driving safer and reduce the number of accidents related to dangerous driving. If the government does in fact implement such a program, we recommend that the program:

• Fine unsafe minibus drivers

While our experiment could not test the impact of a potential fine for unsafe driving, the behavioral economics literature suggests that individuals are roughly twice as likely to avoid losses as they are to seek out gains. Given that losses have stronger effects, this is also likely to ensure that there actually is an overall effect of the program. Importantly, this may offset costs associated with the program.

• Publicize the program in the lead up to implementation

Minibus drivers should be made aware of the program. If they do not know that they could be monitored, the program will be slower to encourage safe driving.

• Select routes for monitoring randomly on a daily basis

This would help prevent drivers from driving artificially safely in order to avoid a fine on a trip when they believe a monitor to be present based on prior information.

• Use few monitors, and change them regularly

The success of such a monitoring program relies on monitors being able to maintain their anonymity. In a country like Georgia, with a small population and dense social networks, maintaining monitor anonymity will be challenging. Hence, the government should consider drawing monitors from one of the civil service agencies with a relatively large staff. The Ministry of Internal Affairs Patrol Police Department would likely be an ideal institution given that patrol police officers are already aware of road safety legislation, and there are a sufficiently large number of officers who could participate in the program on a rolling basis.

The above recommendations are likely to help improve the safety of marshutka driving in Georgia. With less distracted and dangerous driving on Georgia’s roads, the high level of accidents, injuries, and fatalities on roadways are likely to decline. Ultimately, such a policy is likely to lead to safer transport options for passengers.

Safer transport options for passengers: Immediate, lasting, and contagion effects

As the last two blog posts in this series have described, CRRC-Georgia carried out a randomized control trial on minibus safety. This post reports the results of the experiment, and specifically whether there were immediate effects of being monitored, lasting effects on drivers which were monitored, and whether drivers who were not aware that they were being monitored ended up driving safer as a result of contagion effects i.e. drivers talking to each other about the monitoring. Overall, the results suggest that a small minibus monitoring program is likely to decrease dangerous driving behaviors in Georgia.

Did active knowledge that an individual was being monitored matter? To find out, we compare the control group from the first wave with the treatment group observations from the second wave. The statistical analysis suggests that drivers made 1.4 fewer calls per trip, smoked 1.5 fewer cigarettes, made 2.2 fewer illegal passes, and 2 fewer aggressive maneuvers on average. Overall, in the group that was directly aware that it was being monitored, there were 7.2 fewer incidents on average per trip. Other measures such as speed and seat belt non-use did not decline or increase in a statistically significant manner.

Based on these statistics we can conclude that direct knowledge of being monitored lead to fewer distracted and other dangerous driving behaviors. However, did these effects last? To understand whether direct knowledge of being monitored and that one would again be monitored had a lasting effect, we compare the results of the drivers who knew they were being monitored to roughly two weeks later when they did not know they were being monitored. In this case, a lasting effect is present if there was a statistically significant decline in a behavior in the first round of monitoring, and no significant change in the second round of monitoring.

Data analysis suggests that drivers who knew they would be monitored maintained lower levels of smoking, telephone conversations, illegal passing, aggressive maneuvers, and number of incidents overall. However, it is important to note that the drivers did drive less safely than they did when they were aware of being monitored. The differences between waves are however not statistically significant, suggesting that drivers did in fact drive safer roughly two weeks after being aware that they were being observed and would be again.

When it comes to contagion, the results of the statistical tests uniformly show no significant change except for in speed. Given that this is one in ten tests and that there was no significant effect from the first wave of monitoring on speed, we suspect that this test may have been found to be significant based on chance or other extraneous factors.

The table below presents the average treatment effect on the treated for each indicator with Abadie Imbens standard errors in parenthesis. On the line following the indicator name, 95% confidence intervals are provided. One star indicates that the estimate has less than a one in twenty chance of being due to chance alone. Two stars indicates that the estimate has less than a one in one hundred chance of being due to chance alone, and three stars indicates that the estimate has less than a one in one thousand chance of being due to chance alone. No stars indicates no significant difference.

Based on the above statistics we can conclude that direct knowledge of being monitored led to fewer distracted and other dangerous driving behaviors. However, the combination of the lack of significant contagion effect in combination with the increased but not significant increase between treatment groups suggests that a comparison of descriptive statistics is important to gain a sense of whether there was a lasting effect. The table below shows the average of each indicator given above in each group of the experiment.

The average indicator for the third wave of treatment and control group observations are quite similar. The similarity between means suggests one of two things. First, there may have been a small contagion effect since the levels are lower than in the first round control as well as a lasting effect that was not statistically significant. Supporting this conclusion is that both second wave means are lower than the mean in the first observation of the control group. The second possibility, however, is a lack of lasting effect and that an external factor caused the lowering of indicators during the third wave of observation.

Overall, the results of the experiment suggest that direct knowledge of being observed lowers the number of dangerous and distracted driving behaviors by a significant amount. This experience may have a lasting effect on drivers. What does this mean for the policy we proposed in the previous post, and what can the government do based on this information? In the next post, we provide recommendations for the government based on the information presented in this post.

Safer transport options for passengers: A Nudge on Marshutka Safety

Within the auspices of the Safer Transit Options for Passenger’s project, CRRC-Georgia carried out a randomized control trial, attempting to test whether a small policy change could make a large difference to minibus driving safety in Georgia. What was that small change, and how did we measure whether it might matter? This blog post provides an overview of both the policy to be tested and how we measured whether it could work.

The policy

A potentially simple way of decreasing distracted and other dangerous driving practices among minibus drivers is to use anonymous monitoring combined with penalties for dangerous driving. Under such a policy, the government would hire a small number of monitors to ride on randomly selected minibuses throughout the country without informing the drivers. After the ride, the monitor would report on any serious road safety violation as well as the number of distracted driving activities and safety violations carried out. If the driver committed serious traffic violations fines could be given out.

This policy would encourage safe driving and discourage dangerous driving. Moreover, it would require a relatively limited amount of funds from the government. Assuming that only 10 monitors are engaged in the program and they work 200 days a year, they could easily make up to 4000 trips, covering the majority, if not every, minibus route in the country. By randomly assigning monitors to different routes, minibus drivers would not be able to predict whether they are or would be monitored on any given trip. Hence, with the credible risk of being fined, drivers would likely drive safer.

Testing the theory

While the above policy, in theory, is quite sound, practice and theory often diverge. Hence, in order to test whether the policy would in fact be effective, CRRC-Georgia carried out a randomized control trial, which tested whether the knowledge that drivers might be monitored for safe driving and drivers could receive an award might improve their driving in late 2016.

Randomized control trials are a research design that comes from medical research. When doctors are attempting to understand whether a new medicine is effective, they randomly assign whether a patient gets the new drug or a placebo. Randomization is used to try to eliminate whether confounding factors that may make the medicine (in)effective for different individuals are distributed equally between the groups that receive the medicine and the placebo. Following this logic, for the STOP experiment, we randomly assigned minibus drivers to either a treatment or control group.

While in medicine, a treatment is, well a medical treatment, in our case information and action functioned as treatments. Minibus drivers that were assigned to the treatment group were told that they were being observed on a number of driving safety measures, and that the safest drivers would be awarded a gas voucher.  Importantly, as an NGO, we could not make a credible claim about the issuance of fines. Hence, we were unable to fully test the effectiveness of the proposed policy above. Notably, we would expect the issuance of fines for unsafe driving to be a stronger incentive for individuals to drive safer, because people are usually more averse to losses than prone to seeking gains, a phenomenon psychologists refer to as loss aversion.

Between September 20th and October 20th, CRRC-Georgia interviewers observed 360 minibus trips in three waves of observation. In the first wave of observation, minibus routes which had been randomly selected were observed without telling the driver that they were being monitored. This group forms the study’s control group. In the second wave of observation, routes assigned to the treatment group were observed. In the third wave of observation, observers returned to both the control and treatment minibuses for anonymous observation.

Minibus drivers in the treatment group who drive along similar routes were informed that:

1. Their trip would be monitored for safety along a number of dimensions;
2. A monitor would return in the coming weeks and monitor their driving as well as other drivers again without telling them;
3. If they were found to be among the safest drivers, they would be rewarded with a petrol voucher.

Over the course of the trip, monitors in all three waves recorded how many times drivers:

1. Smoked;
2. Text messaged;
3. Had telephone conversations;
4. Did not wear a seat belt;
5. Passed in areas it was not legal to do so;
6. Made other aggressive driving maneuvers;
7. Behaved aggressively towards passengers;
8. Behaved aggressively towards non-passengers.

Monitors also recorded stop and travel time, whether additional seats were added to the bus, whether passengers stood during the ride as well as a number of characteristics about the state of the minibus. Following the trip, the average speed of travel was calculated.

The above research design allows for a number of comparisons. First, by comparing the first wave control group to the second wave treatment group, we can test how the direct knowledge that one is being observed for safe driving effects driver safety. Second, by comparing the second wave treatment group to the third wave treatment group, we are able to tell whether the knowledge that one might be monitored again in the near future would have some lasting effect. Third, by comparing drivers in the first wave control group to drivers in the third wave control group, we can tell whether there was a contagion effect from the experiment i.e., whether the treated drivers talked to the non-treated drivers about the monitoring and they in turn also became safer drivers.

To test for the above types of effects, we used multivariate matching with genetic weights and calculations of average treatment on the treated (ATT). For readers interested in these methods, please see here. Using these methods, we calculate both the size of an effect and the probability that it emerged by chance alone.

In the next post in this series, which will appear tomorrow, we report the results of the experiment.

Safer transport options for passengers: Distracted and dangerous driving among minibuses

Over the course of this week, CRRC-Georgia will publish the results of a randomized control trial on minibus safety. While the introduction post to this series highlighted that Georgia’s roadways are dangerous, just how dangerous minibus drivers are has largely been left undescribed. As part of the Safer Transport Options for Passengers project, CRRC-Georgia collected data on dangerous and distracted driving practices on minibuses. This blog post reports descriptive statistics about distracted and dangerous driving from drivers unaware that they were being monitored.

While estimates do not exist for Georgia for how many accidents are caused by distracted and other dangerous driving practices, they are very likely contribute to the high fatality rates on the roads in Georgia. When it comes to distracted driving alone, studies from other contexts suggest that a distracted driver’s chances of being in an accident are four times higher. Cell phone use is associated with increased incidence of accidents among both novice and experienced drivers. Importantly, commercial vehicle drivers are no exception, with increased risk of accident associated with distracted driving among commercial drivers.

Overall, among minibus drivers unaware of being monitored, 96% engaged in some form of poor driving behavior, with illegal passes being the most common, followed by making telephone calls, and other aggressive driving maneuvers. Notably, few drivers were observed text messaging while driving. While we cannot claim that the data presented here is representative, for technical reasons, it is highly suggestive of the high prevalence of dangerous and distracted driving practices among minibuses on Georgia’s roadways.

The graph below presents the average number of times a driver engaged in a dangerous driving behavior as well as the maximum value for each indicator. Although the minimum value was 0 for each indicator, the high values for the maximums suggest that some drivers are particularly prone to dangerous driving, with as many as 62 dangerous events observed in a single trip.

The above statistics suggest that distracted and dangerous driving are common problems among minibus drivers in Georgia. In order to help ameliorate the situation, CRRC-Georgia tested whether a simple minibus monitoring policy would decrease the prevalence of dangerous and distracted driving. In subsequent posts in this series, we report the results of the randomized control trial, which suggest that such a program would in fact be effective.