Case Study 1: Use of the Analytic Failure Simulation Technique in elections forecasting

Election forecasting is a core occupation of policy analysts all over the world.

Analytic Failure Simulation is a structured analytic technique that can increase the accuracy of estimates and as a consequence raise analyst credibility.

Election forecasting can also be great fun. I’ll show you how. Judge for yourself.

Early parliamentary election in Austria was agreed by the ruling left-centrist coalition on 15 May 2017 and held five months later.

The ruling leftist Socialist Party and the centrist People’s Party reached the consensus to hold early elections following several months of political deadlock, which all but paralysed government and parliamentary activity.

The junior coalition partner – People’s Party was fighting for its relevance and political survival. Its role in the government was dwarfed by its partner.

However, popularity and policies of the powerful and quasi institutionalised Socialist Party were severely dented by the refugee crisis of 2015. Over the span of two years it led to an increase in crime, proliferation of ethnic gangs, a great burden placed on social security and health insurance schemes, and a growing popularity of the right-wing Freedom Party.

The People’s Party became acutely aware of a clear and present danger of an imminent collapse of a system set up during four decades of socialist rule. While the socialists might have had some chance in surviving a political tsunami, it could have easily swept the centrists off the board.

In an internal flash coup, both risky and daring, the young Austrian Foreign Minister succeeded in side-lining older-generation politicians in the Popular Party. He then proceeded with reshaping the People’s Party into a vertical hierarchy with himself firmly at the top.

The Austrian public cheered the audacity of the young politician.

In less than one week, the popularity of centrists doubled. The Freedom Party also welcomed the call for election. It entered the electoral campaign as the most-prepared and well-organised political force. Not seriously planning to win, it could nevertheless only gain from either the socialist or the centrist victory. It was all set for taking the second place – and playing the role of the kingmaker.

Election forecasting under these circumstances offered a great opportunity for practicing the structured technique of Analytic Failure Simulation.

I haven’t got any experience in election forecasting worth mentioning.  I am certainly not a “Superforecaster”, in the parlance of Philip Tetlock. As a matter of fact, I have never taken my ability to produce any kind of estimates with any seriousness at all.

Moreover, I tend to agree with the view expressed by Nassim Taleb in “The Black Swan” regarding the futility of forecasting. Taleb stipulates that there is no way, and no tool, to overcome the uncertainty inherent in the evolution of real-world issues.

Minute random changes in event chains change futures every minute. Future is random, and randomness cannot be predicted. Alternative futures have become so complex and information noise so loud that tracing the most probable path into an even modestly distant future has become as good as impossible.

But what the heck, I still wanted to give it a try.

As stipulated by Richards J. Heuer Jr. et al., the technique of Analytic Failure Simulation aims to challenge the accuracy of a conclusion regarding futures analysis.

The starting point: your preferred hypothesis /prediction/recommended action has failed. It is an accomplished fact – now explain why.

The logic of the method is quite simple.

The admission of failing in my forecast released me from the grip of some of my most powerful biases. There was no need any more to defend my viewpoint, which among other things would have required hiding its flaws.

I could discard my conclusions and view them clinically from a comfortable distance. Since I didn’t have any sense of ownership in them left, I could look at them with a cold eye. Better than that, I became quite excited about cutting the corpse of may deceased prediction up to look in its dark corners for reasons it became such a failure.

The application of Analytic Failure Simulation has one downside. Well, sort of. It requires time, discipline and mental effort – precisely because it is structured. This is also true for most other structured analytic techniques. Come to think of it, it is more of an upside. There is no way it will allow you to cut corners.

Taking Analytic Failure Simulation the full circle involves addressing the following issues:

  • Were my key assumption not valid?
  • What alternative hypotheses I didn’t consider?
  • What external influences affected the outcome?
  • Did I not sufficiently consider deception?
  • Were some of the sources that produced key evidence unreliable?
  • Did I ignore any counterfactual evidence?
  • Was I misled by absence of information?

election forecasting using analytic failure simulation

My initial forecast came out as follows: 32-28-19

The Socialist Party will win the election without breaking a sweat with the People’s Party finishing second within 5% and the Freedom Party taking the third place by falling below 20% of the vote.

My initial estimate was backed by the following arguments:

  • The belief of Austrian voters in the socialist political establishment has become deeply ingrained in their mind-set and will be very resistant to change.
  • Short-term forecast of Austrian economic growth has improved from moderate to record 2,7% (the best in the EU). It is very rare that in times of economic stability and growth voters opt for political change.
  • The migrant crisis has subsided. The number of migrants entering Austria has halved, it was expected that only about 8,000 will be granted asylum in 2017 (with the cap set at 34,000). This was a big feather in SPÖ´s cap making voters confident in their crisis management skills.
  • Supporting soft EU stance on migrants will anchor left-centrist voters. Recent tough speak on border controls and a tougher stance vis-a-vis Turkey on the migrant issue will stop voter drain to the Freedom Party.
  • The campaign of the Socialist Party focused on social issues – pensions, minimal pay, lowering taxes. It succeeded in touching the public nerve.
  • Vienna is the key to winning the election since it accounts for 18,5% of the national vote. It has remained under firm socialist control for over four decades. The current Chancellor is an ex-top business manager. He will be able to command support from main industrialists and corporations located in Vienna. As a result, Vienna will back the Socialist Party by a solid margin which will become the basis of a nation-wide victory.
  • When real power is at stake, nobody has a chance against the “red machine”. Failure is not an option as it would expose skeletons in the cupboard accumulated during decades of socialist rule. Painted into a corner, the Socialist Party will fight like a tiger – and win.
  • The candidate of the People’s Party will put up an impressive show. But his rhetoric will deflate when it came to a debate on economic, labour and social issues, in which he had no experience.
  • The Freedom Party was robbed of its stand on migration. And it had nothing else left to capture voters’ imagination with.

Satisfied with the apparent forcefulness of my arguments and key assumptions, I put the subject out of my mind for a couple of days to let the reasoning circuits cool down a bit.

Then I flipped the toggle.

And took it for a fact that the election was won by the People’s Party.

And grabbed the scalpel. Now, where could my beautiful initial analysis be flawed?

True, Austrian voters were groomed by four decades of socialist rule.

But that is precisely the reason why they have grown disappointed with the traditional way of doing politics. The bucket overflowed. They wanted a new start. The precedent set by Macron in France showed that that was possible. The People’s Party with a facelift provided the means to achieve a similar end.

True, the migrant crisis has (temporarily) subsided. But it left a huge mess in its wake.

An explosion in street crime and gang wars had a sickening effect particularly on city voters who used to provide stalwart support to socialists.

Several cases of embezzlement of public funds allocated to refugee management came to light, uncovering widespread corrupt practices.

Migrants displayed no intention of integrating themselves into the Austrian way of life, only demanding social security money and to be left alone.

In response, the new People’s Party dynamically repositioned itself by shifting far to the right of the political spectrum.

This move helped to hunt two birds with one stone. It responded to the shift in popular perceptions and expectations. And at the same time it took a bit of wind out of the sails of the Freedom Party. On issues related to border control and migrant management the centrists became “more pious than the Pope”.

True, socialists’ focus on social issues attracted voter attention.

But for the majority of voters, the key election issues were uncontrolled migration and creeping islamisation. There, socialists had little to offer. Besides, both the People’s Party and, above all, the Freedom Party had fairly competitive programmes in this field. They also enjoyed considerably greater credibility among voters.

True, Vienna is a key electoral battleground. But it is only one of the two.

The province of Lower Austria accounts for some 20.1% of the national vote. And historically, it votes People’s Party. Its candidate could win overall if he captured Lower Austria by a wide margin and lost to the ruling socialist Chancellor in Vienna by a narrow one.

Not an impossible feat. During the electoral campaign the youthful élan and untainted reputation of the centrist Foreign Minister received very warm response from Viennese voters. Particularly,  the younger generation of civil servants and rising stars of corporate business. They saw a sudden chance of taking a shortcut that could save them up to a decade of boring ascent up the career ladder.

True, when painted into the corner, the Socialist Party fought like a tiger.

But that proved to be not enough. The Tal Silberstein affair publicly exposed socialists’ dirty linen, and that looked real ugly.

In a campaign of personalities, the People’s Party candidate was easily everybody’s darling. His political ranking has been consistently high for close to two years. And his new positioning as a moderate rebel for common sense was understandable and appealing to the voters. Besides, his newly formed political organization was free from scandal or bad news.

Last but not east, the country-wide shift to the right also gave a powerful lift to the Freedom Party.

True, it had to yield some territory to the People’s Party on migration and refugee management issues. But it could still cash in on its biggest asset – credibility among voters.

The whole agenda of the Freedom Party, which it promoted during the better part of the last decade, overnight became political mainstream.

Over the years, its consistent positioning has helped to build up a power base of loyal voters.  While the People’s Party could lure away a good chunk of socialist voters, those of the Freedom Party stayed mostly loyal.

Accordingly, my revised intuitive estimate of the election outcome was People’s Party 33%, Freedom Party 27% and Socialist Party 25%.

I followed very simple logic – no winner would get more than 35% and no third place less than 20%. If the People’s Party wins, putting them at 33% will cover both the upper and the lower margin of probable accuracy as there was every reason to expect that they’ll pass the 31% threshold.  While a 2% error in an estimate may not be something to boast about, it is nothing to be ashamed about either.

When estimating the result of the Freedom Party, I used a slightly different reasoning to choose the upper boundary in a range of possibilities. While my estimate of the winning result carried minimal risk of being significantly wrong, my estimate of the runner-up performance was realistic but high-risk.

This approach generally follows the strategy for making predictive estimates favoured by Taleb, for whose judgement I have great respect.

Finally, if the socialists rank only third, it seems unlikely that they sink all the way to the 20% bottom. There is still quite some life left in them.

My new arguments looked fairly convincing to me. But in all frankness, so did those underpinning my initial analysis. That is, before I gutted them.

So, my next step was to generate my own data to see which of the two scenarios it would refute.

According to the taxonomy developed by Richards J. Heuer Jr. et al, structured analytic techniques form one of four analytic domains. The other three are unaided expert judgement, quantitative analysis of expert-generated data and quantitative analysis of empirical data.

It is a great idea to use some mix of those, as they can mutually enhance each other through feedback loops.

Generating your own data is the next best thing after ice cream.

It can back your conclusions with the solidity data tends to project. And as a consequence, boost both your confidence and credibility. Besides, it can make your estimates more accurate, too, which should be your ultimate objective, after all.

In my case, it was as easy as one-two-three.

I looked up the results of the Austrian presidential election in December 2016. They were available in minute detail, down to base electoral districts. I was satisfied with using results data for provinces and provincial capitals.

With some sleight of mind I calculated the true voter numbers. Then I made small adjustments to account for new voters and discount for somewhat lower rates of absenteeism that formed part of my estimates.

All in all I had 114 data points to fill.

Which I did, taking into consideration various contributing factors. Such as the popularity of socialist Minister of Defence in his native province of Burgenland. Or support from communities in the vicinity of the border with Italy at Brenner pass to the right-wing proposals of strengthening border control. Too, the high level of popularity enjoyed by the candidate from the People’s Party in cities with large universities.

The outcome wasn`t really surprising.

It tended to refute my initial analysis in favour of the estimate achieved through analytic failure simulation.

People’s Party – 30,48%; Socialist Party – 26,11%; Freedom Party – 25,70%.

election forecasting with analytic failure simulation

I completed my estimates around 17 August 2017 and forgot all about them. The purity of my experiment shunned recourse to estimate updates.

When the official election results were announced, I was stunned.

My estimate was within 1% for all three top-placed participants and correctly reflected their actual ranking, too.

People’s Party – 31,47% (me off by 0,99%); Socialist Party – 26,86% (off by 0,74%); Freedom Party – 25,97% (off by 0,27%).

election forecasting more accurate using analytic failure simulation

Did it make me a “Superforcaster”? Hell, no.

It just showed me that using the right structured analytic technique did result in a significant improvement in the accuracy of my predictive estimates.

It may be interesting to observe that my fairly accurate estimate was generated by using wildly inaccurate data.

Out of 114 data points, only 21, or 18,4% were estimated within 1% deviation from the actual result.

That, however, does not subtract from the validity of the technique involving generation of your own data.  Individual estimates may be way off the mark. Some will inevitably turn out higher and other, lower. But spikes and troughs tend to cancel each other out. And given a sufficiently large number of data points, the aggregate estimate has a good chance of being reasonably close to the true result.

Since then, I had a similar experience with other data sets I generated. As a consequence, I have developed a firm belief in the validity of this method.

For example, it stands up to the challenge when using Force Field Analysis in conjunction with the Futures Wheel technique. Its objective is to suggest the Most Probable Path, which involves generating some 280 data points.

It also does remarkably well when grading student homework on my course. While being a considerably less complex exercise it nevertheless involves some 35 independent data points for each student.

Any single one of these might or might not agree with each of the 35 grades I gave them.

But none of some 250+ students that I have taken through the meat-grinder so far have contested their aggregate course grade. They accepted the method behind it was fair.

Leave a comment

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.
You need to agree with the terms to proceed