[Note: In order to help monitor the fidelity of the October 2016 parliamentary election results, CRRC-Georgia will carry out quantitative analysis of election-related statistics using methods from the field of election forensics within the auspices of the Detecting Election Fraud through Data Analysis (DEFDA) project. The Project is funded by the Embassy of the United States of America in Georgia, however, none of the views expressed in the following blog post represent the views of the US Embassy in Georgia or any related US Government entity.]
On Friday, August 19th, CRRC-Georgia presented and published a pre-analysis report for the Detecting Election Fraud through Data Analysis project, which contained analysis of the new electoral boundaries set up following the 2015 constitutional court ruling that the previous boundaries were unconstitutional.
The report also demonstrated how the methods of statistical analysis that CRRC-Georgia will use to monitor the 2016 elections work in practice. To do so, we used precinct level data from the 2012 party list elections. Specifically, CRRC-Georgia carried out two types of statistical analyses:
While Monday’s blog shows the logical checks that CRRC will apply to the final CEC vote records, today we discuss the tests used to identify statistical anomalies in vote counts.
Election Forensics: Detecting statistical anomalies in voting data
Direct observation of polling stations is the best method available to ensure the accuracy of the vote, however, election observers cannot be everywhere all the time. Given this fact, the field of election forensics, a subfield of political science, has developed a number of statistical tests to look for statistical anomalies in election returns, which may suggest suspicious election-related activity. Although a number of rather complicated statistics exist, we focus on a number of simpler tests. Specifically, we use tests based on the distribution of the second digit in the number of votes cast, the final digit in the number of votes cast, and the distribution of turnout within an electoral district.
Second digit tests are based on Benford’s law. Benford’s law provides the expected probability of the first digit being any digit one through nine in a number with multiple digits. Although one might expect this number to be equally likely to be any number, in fact 1 is more likely than 2, 2 more likely than 3, etc. Using Benford’s Law, accountants test various documents for anomalies that may suggest issues in documents. This law also applies to the second digit in a number, which researchers have found is more suitable for testing election results. A similar logic is applied to elections as in accounting, and in this blog, we specifically test whether the skew, kurtosis, and the average of the second digit and its distribution follow the expected distribution or not. Instances of non-conformity to Benford’s law may suggest electoral malfeasance.
Besides second digit tests, a number of tests have been proposed for the last digit in vote counts. Here, the expected distribution of digits is much more intuitive, and one expects each digit, zero through nine, to be approximately 10% of the total distribution. Based on this distribution, we test the mean of the last digit and of the mean of the count of zeros and fives in the final digits of votes.
In order to test whether the above noted digit tests in fact indicate potential issues or whether the difference between the observed and expected values was a chance variation, we use a statistical method called bootstrapping. This method lets us to estimate 99% confidence intervals. In the present case, the confidence intervals provide a range within which the result could have fallen by chance. If the range covered does not include the expected value for a given test statistic, we conclude with 99% confidence that the number is different not by chance alone.
Finally, voter turnout is expected to have a relatively normal distribution with a single mode. Based on this expectation, we test whether voter turnout in each electoral district has a single mode or multiple modes using what statisticians refer to as a dip test.
Before reporting the test results, it is worth noting several important caveats when interpreting these tests:
In total, 11 districts show statistical anomalies in the test results, and a total of 15 tests report suspicious results. Results are presented in Table 5. In the rows with district names and numbers, the actual test values are reported. In the row below the district name, 99% confidence intervals are reported. Red cells in the table indicate the presence of a statistical anomaly.
Rustavi’s electoral returns set off three statistical tests. Given that we have no reason to expect specific voting patterns in Rustavi compared to other areas in the country that did not set off suspicious tests, this suggests that there may have been electoral malfeasance in Rustavi in 2012. Reviews of election monitoring reports, however, did not suggest electoral malfeasance. This test may be picking up on undetected electoral malfeasance from 2012 in Rustavi. Although unlikely, these three tests could have also been set off by chance.
In Kobuleti, two tests were also set off. In Kobuleti, we would not expect a particularly distinctive voting pattern. Hence, there is a relatively strong reason to believe that electoral malfeasance may have occurred in Kobuleti in the 2012 elections. This contention is supported by election monitoring reports, which reported issues in Kobuleti.
In Bolnisi, two tests were set off. Complaints were filed in Bolnisi on election day, and the test may have been set off by these issues. However, given Bolnisi’s relatively high ethnic minority population and distinctive voting pattern, the tests could have been set off by this rather than malfeasance.
Eight other districts had single positive tests for electoral malfeasance, including Vake, Saburtalo, Kareli, Akhaltsikhe, Adigeni, Vani, Senaki, and Martvili. A review of the OSCE and GYLA election monitoring reports suggest that issues may have occurred in at least half of these districts. Although these positive tests could have occurred by chance alone, the four districts in which a test was set off and observers did not report malfeasance in may also suggest unreported problems in the 2012 elections.
This blog post has described the methods CRRC-Georgia will use to detect statistical anomalies in election returns. For more on the methods CRRC-Georgia will use to monitor the elections, see our pre-analysis report, here, and take a look at Monday’s blog post on logical inconsistencies in election records.
On Friday, August 19th, CRRC-Georgia presented and published a pre-analysis report for the Detecting Election Fraud through Data Analysis project, which contained analysis of the new electoral boundaries set up following the 2015 constitutional court ruling that the previous boundaries were unconstitutional.
The report also demonstrated how the methods of statistical analysis that CRRC-Georgia will use to monitor the 2016 elections work in practice. To do so, we used precinct level data from the 2012 party list elections. Specifically, CRRC-Georgia carried out two types of statistical analyses:
- Logical checks of official election returns, which test whether there were data entry errors when the vote was being recorded and collated;
- Tests for statistical anomalies in the official electoral returns, which may suggest electoral malfeasance.
While Monday’s blog shows the logical checks that CRRC will apply to the final CEC vote records, today we discuss the tests used to identify statistical anomalies in vote counts.
Election Forensics: Detecting statistical anomalies in voting data
Direct observation of polling stations is the best method available to ensure the accuracy of the vote, however, election observers cannot be everywhere all the time. Given this fact, the field of election forensics, a subfield of political science, has developed a number of statistical tests to look for statistical anomalies in election returns, which may suggest suspicious election-related activity. Although a number of rather complicated statistics exist, we focus on a number of simpler tests. Specifically, we use tests based on the distribution of the second digit in the number of votes cast, the final digit in the number of votes cast, and the distribution of turnout within an electoral district.
Second digit tests are based on Benford’s law. Benford’s law provides the expected probability of the first digit being any digit one through nine in a number with multiple digits. Although one might expect this number to be equally likely to be any number, in fact 1 is more likely than 2, 2 more likely than 3, etc. Using Benford’s Law, accountants test various documents for anomalies that may suggest issues in documents. This law also applies to the second digit in a number, which researchers have found is more suitable for testing election results. A similar logic is applied to elections as in accounting, and in this blog, we specifically test whether the skew, kurtosis, and the average of the second digit and its distribution follow the expected distribution or not. Instances of non-conformity to Benford’s law may suggest electoral malfeasance.
Besides second digit tests, a number of tests have been proposed for the last digit in vote counts. Here, the expected distribution of digits is much more intuitive, and one expects each digit, zero through nine, to be approximately 10% of the total distribution. Based on this distribution, we test the mean of the last digit and of the mean of the count of zeros and fives in the final digits of votes.
In order to test whether the above noted digit tests in fact indicate potential issues or whether the difference between the observed and expected values was a chance variation, we use a statistical method called bootstrapping. This method lets us to estimate 99% confidence intervals. In the present case, the confidence intervals provide a range within which the result could have fallen by chance. If the range covered does not include the expected value for a given test statistic, we conclude with 99% confidence that the number is different not by chance alone.
Finally, voter turnout is expected to have a relatively normal distribution with a single mode. Based on this expectation, we test whether voter turnout in each electoral district has a single mode or multiple modes using what statisticians refer to as a dip test.
Before reporting the test results, it is worth noting several important caveats when interpreting these tests:
- Test results are probabilistic, which means that they say the distribution is highly unlikely (would occur 1% of the time in the present case), rather than impossible to occur in the absence of issues. For the tests, we calculated 99% confidence intervals. With 99% confidence intervals and having conducted 444 tests on the 2012 proportional election results, statistically we would expect between four and five tests to be set off in the absence of issues due to chance alone.
- The lack of a test being set off does not necessarily mean a problem occurred, but it does suggest the need for further examination;
In total, 11 districts show statistical anomalies in the test results, and a total of 15 tests report suspicious results. Results are presented in Table 5. In the rows with district names and numbers, the actual test values are reported. In the row below the district name, 99% confidence intervals are reported. Red cells in the table indicate the presence of a statistical anomaly.
Rustavi’s electoral returns set off three statistical tests. Given that we have no reason to expect specific voting patterns in Rustavi compared to other areas in the country that did not set off suspicious tests, this suggests that there may have been electoral malfeasance in Rustavi in 2012. Reviews of election monitoring reports, however, did not suggest electoral malfeasance. This test may be picking up on undetected electoral malfeasance from 2012 in Rustavi. Although unlikely, these three tests could have also been set off by chance.
In Kobuleti, two tests were also set off. In Kobuleti, we would not expect a particularly distinctive voting pattern. Hence, there is a relatively strong reason to believe that electoral malfeasance may have occurred in Kobuleti in the 2012 elections. This contention is supported by election monitoring reports, which reported issues in Kobuleti.
In Bolnisi, two tests were set off. Complaints were filed in Bolnisi on election day, and the test may have been set off by these issues. However, given Bolnisi’s relatively high ethnic minority population and distinctive voting pattern, the tests could have been set off by this rather than malfeasance.
Eight other districts had single positive tests for electoral malfeasance, including Vake, Saburtalo, Kareli, Akhaltsikhe, Adigeni, Vani, Senaki, and Martvili. A review of the OSCE and GYLA election monitoring reports suggest that issues may have occurred in at least half of these districts. Although these positive tests could have occurred by chance alone, the four districts in which a test was set off and observers did not report malfeasance in may also suggest unreported problems in the 2012 elections.
This blog post has described the methods CRRC-Georgia will use to detect statistical anomalies in election returns. For more on the methods CRRC-Georgia will use to monitor the elections, see our pre-analysis report, here, and take a look at Monday’s blog post on logical inconsistencies in election records.
No comments:
Post a Comment