10 August 2016

Opinion polls are broken; time for something new

By Hugo Winn

Public opinion polls are everywhere. They dominate our political cycles and to a degree structure the very nature of our political debate.

They have huge implications for who is elected and who is not. During this year’s Republican primaries Fox news announced that, in order to participate in its first prime-time debate, candidates had to “place in the top ten of an average of the five most recent national polls.” Where the candidates stood on the debate stage would also be determined by their polling numbers.

The polling industry is now a multi-billion dollar industry. Between the late 1990s and 2012, twelve hundred polling organizations conducted nearly thirty-seven thousand polls. And despite obvious public failings in the last few years, the popularity of opinion polling shows no sign of waning.

The modern public opinion poll has existed in an almost unchanged state since the 1930s when George Gallop introduced the idea of surveying mass public attitudes.

Gallop’s innovation was to call the landlines of thousands of American’s who evenly represented the demographic makeup of America. At the time response rates – the percentage of people providing their views when called – were around 90 percent; American’s were un-cynical of pollsters and typically had a landline which pollsters could easily call them on. Over time, both of these factors have rapidly declined; today an average US response rate is in the single digits.

This has massive implications for the selection bias of polls. As responses became more difficult, the challenge pollsters faced in creating evenly representative samples through telephone polling became exponentially harder.

This trend is irreversible. Fewer and fewer voters now have landlines, and mobile phone public opinion calling is generally illegal in the West. Last year, Gallop Polls was forced to pay millions to voters it had telephoned on their cell phones between 2009 and 2013.

In response, polling companies are shifting towards Internet polling. But, again, tight regulation on unsolicited contact means pollsters must wait for internet users to contact them, making building representative samples nigh on impossible.

The failure of public opinion polling is global in scope: In March 2015, Israeli polls completely failed to predict Benjamin Netanyahu’s victory. Two months later every major national poll in the UK failed to forecast the Conservative Party’s victory. A year later, every US polling agency failed to predict Donald Trump would win the Republican Party nomination. Following the failure of opinion polls to predict a Brexit victory earlier this year, investors and businesses lost over a billion pounds from currency devaluation, within days.

But, something remarkable also occurred during the Brexit vote. Shortly prior to the referendum a number of major Hedge funds including Bridgewater Associates and Brevan Howard Asset Management began testing a new predictive tool for data mining on social media.

Unlike previous, largely academic, studies of social media opinion analysis, the firms used in-house Artificial Intelligence algorithms, developed for hedging risk, to understand voting intention amongst online voters on a massive scale.

Following the vote Brevan Howard Asset Management gained 1 percent on its $16 billion macro fund. Hedge funds globally lost 1.6 percent.

Pollsters have traditionally sneered at the possibility of accurately measuring public opinion by mining social media data. Their concerns are twofold. First, they believe social media is woefully biased as a dataset. They are right. In the UK, about 33 million people are on Facebook, but the number actively posting political opinions and views on Twitter (the easiest channel to analyse) is much lower. However, discounting such a rich data-set on this count seems unnatural. Instead, the challenge should be to analyse samples on social media in exactly the same way opinion polls analysed telephone records, weighting by all the markers Gallop first proposed back in the 1930s.

Second, pollsters believe sentiment analysis – the measurement of how positive or negative someone online is about something – is far too unsophisticated to accurately predict voting intention. On this count they are wrong. Over the last few years, a quiet revolution has been taking place in A.I. enabled language analysis. This revolution is subtle but has huge ramifications for opinion gathering. Modern algorithms are increasingly capable of understanding the nuances of language using a technique called Natural Language Processing and can increasingly learn the individual way people communicate about specific issues. This ‘deep learning’ is fascinating and potentially highly disruptive.

In September this year, a team of UK and US political researchers, data scientists, and mathematicians – including myself – will come together to test out the cutting edge of this technology. The project, called Deep Listen, is a bold attempt to predict an election outcome using artificially intelligent algorithms, representative social media samples, and predictive modelling. It will first be used to predict the outcome of the UK Labour Leadership Election.

Behind the scenes pollsters concede that they are less sure how to conduct good survey now than four years ago, and much less than eight years ago. As we approach a new round of elections, with global ramifications, new predictive exercises are fundamentally important to explore if we are to accurately keep our finger on the world’s pulse.

Hugo Winn is Founder of Deep Listen; an A.I. project to make intelligent predictions.