A (not so) quick guide to polling in the 2010 election

You know that election season is close when the weekend papers begin to fill their front pages with talk of ‘swings’ and ‘top-line figures’, whilst those in Westminster begin to talk in reverent tones of those quasi-mythical oracles of modern politics: the pollsters. But the art of reading political tea leaves is a difficult one. With different polling companies often producing divergent results, and different analysts offering conflicting opinions on what each poll means, it can often be difficult to know who to believe. This guide is intended to help you navigate the treacherous waters of the opinion poll, and equip you with the knowledge required to make your own informed judgements.

A very short introduction:

Polling is usually carried out by a specialist company, more often than not under the direction of a commissioning organisation, such as a newspaper, political party or pressure group. Most polling companies will have their own methodology for gathering responses and selecting a representative sample for analysis, although the questions asked vary depending on the demands of the commissioner. The largest polling companies in Britain are YouGov, ICM, Ipsos-MORI, Populus and ComRes, although the field contains a number of newcomers since the last general election, including BPIX, Angus Reid and Harris. Many national newspapers have long-standing relationships with a particular polling company, such as The Guardian’s with ICM, The Independent’s with ComRes and that between YouGov and The Sun and The Telegraph.

Methodology:

Polling is an inexact science which can often produce contrary results. For example, Angus Reid polls have consistently shown a Conservative lead of between 12 and 16 points since the turn of the year, whilst YouGov has placed the Tory lead between 2 and 12 points over the same period. What can account for this difference?

At heart, of course, polling companies all have a common objective: to gather opinion from a representative sample on a given topic in order to highlight underlying trends. However the ways in which the pollsters approach this goal differs from company to company.

The first major dividing line is drawn in the ways in which different pollsters gather their data. Certain companies, including YouGov, Harris and Angus Reid, use online surveys completed by a registered panel to form their sample. Other companies, such as ComRes and Populus, use telephone interviews, and a declining number still use face-to-face interviews, including TNS BMRB. Still others, ICM included, use a mix of online, telephone and face-to-face interviews to inform their polls. Ipsos-MORI switched from a face-to-face methodology to a telephone-based research method after the London mayoral election of 2008, citing criticisms that the face-to-face technique gave disproportionate weighting to public-sector workers.

A more complicated difference is found in the ways in which different companies ‘weight’ their samples. Weighting is the process through which the opinions collected in a poll are exaggerated or underplayed depending upon the stated allegiances and demographics of the respondent, in order to bring the sample in line with the nationwide equivalents, thus producing a ‘representative sample’. All pollsters do this, but the factors that are chosen to weight the sample vary from company to company.

Most companies will use demographic data - such as age, race, sex, income, and employment status - collected from their panel to weight their responses. This helps to iron out any anomalies that might be thrown up by their sampling methods. For example, it would probably be accurate to assume that a face-to-face survey conducted during the daytime might catch a disproportionately high number of unemployed people, as those in work are not available to be questioned. Equally, online surveys might over-represent higher income groups, as they might be presumed to have better access to the requisite technology.

Given that this sort of weighting is fairly standard however, it is unlikely that this is a major source of divergence between polling companies. More contentious is the use of political data as a weighting factor. Political weighting – which uses factors such as party membership, voting history and newspaper readership to balance the sample – is applied differently by most polling organisations, and has evolved in order to address discrepancies observed between the data collected by pollsters and historical actualities. Political weighting became widespread after pollsters suffered a disastrous 1992 General Election campaign, where predictions of a Labour victory were all but unanimous across the companies. A subsequent Market Research Society (MRS) inquiry found that respondents were more likely to refuse to reveal their voting history and intention if they associated themselves with ‘unfashionable’ parties (a phenomenon known as the ‘Spiral of Silence’ in polling circles). Another common problem is deciding upon how to include the opinions expressed by those who are uncertain whether or not they will vote; they often represent a significant proportion of the sample, but their responses are clearly not as valuable as those who are certain of their intention to vote.

The methods used to address these challenges vary depending on the polling organisation. Populus, ComRes and ICM all use a respondent’s vote at the last election to determine their weight in relation to voting intention. ICM, MORI, ComRes and Populus also all use the stated likelihood to vote as a weighting factor, whilst YouGov and Angus Reid factor in newspaper readership.

However weighting on these terms is not as simple as it might seem. Statisticians have noticed an undeniable tendency for ‘false recall’ amongst a significant proportion of any given survey panel. False recall is the collection of inaccurate data due to the respondent’s inability to remember their own voting history, or their desire to distort their previous voting record. A good example of the latter is a hypothetical Conservative supporter, who voted tactically for the Lib Dems at the previous election, but wishes to express their support for the Tories. False recall can therefore also compound the differences between survey methods, as online polling can store data pertaining to an individual’s responses over long periods without fear of inaccuracy, whereas telephone and face-to-face interviews rely to a much greater extent upon the honesty and memory of their respondents. ICM, Populus and ComRes all adjust their survey results to account for false recall. Furthermore, all polling companies test their methodologies against historical data.

A final methodological point to bear in mind is the order in which different pollsters place their questions. Although notoriously difficult to quantify, research has shown that responses to a given question can be shaped by whether it is placed at the beginning, middle or end of a survey. Of particular note in this respect is the placement of the voting intention question. All companies aside from Angus Reid ask this ‘headline question’ first, which could explain why Angus Reid has returned such divergent results.

So who should I believe?

The somewhat predictable response to this commonly-asked question is that there is no simple answer.

However in the two most recent national elections, YouGov has had the most success by final polling, judging the eventual 2008 London mayoral result perfectly when most other pollsters had predicted a victory for Livingstone, and getting the 2009 European election result to within an average margin for error of 1%, compared to second-placed Populus’ 1.6%.

Past form is not a fail-proof yardstick for measuring likely success in the up-coming election however, for a couple of reasons. Firstly, YouGov, like most other polling companies, have made tweaks to their methodology which has not yet been tested against a real election result. The different constituencies and boundary changes can also disrupt the local weighting done by pollsters in a general election campaign, especially where this changes the rules of engagement for tactical voters.

It’s also worth noting that a general election campaign is a different beast to the local and European equivalents, where shifts in focus between emerging policies, scandals and intrigue can produce a much more pronounced swing in the polls. Having duly noted this, YouGov has nevertheless been widely credited with producing the most accurate voting intention figures ahead of the 2001 and 2005 elections.

Ipsos-MORI’s policy of only including respondents who describe themselves as certain to vote in their samples mean that their top-line figures are often more volatile than other pollsters. Although their most recent return of a 5 point Tory advantage is broadly in-line with other polls, since 2008 they have recorded a 28 point lead for the Conservatives, and have seen successive month-on-month swings of 11 points over the last six months.

However MORI did enjoy a very successful 2005 election, getting the Conservative and Liberal Democrat vote shares spot on, and missing both Labour and Others by 2 points. Similarly, Populus correctly predicted Labour and Lib Dem vote shares in their final 2005 poll, placing the Conservatives to within a 1% margin of error. ICM has historically overstated Labour support during general election campaigns.

A word of caution:

You can, to paraphrase George Canning, prove anything with statistics except the truth. Many an over-enthusiastic commentator has eagerly extrapolated headline-friendly conclusions from opinion polls, only to find that on closer inspection the detail does not bear them out. The pitfalls in polling are numerous – here are some of the more common snares to avoid.

The golden rule for analysts is to make sure that in charting the fortunes of parties through the polls, one only ever compares like for like. Given the differences in methodology outlined above, it would be extremely misleading to describe, for example, Labour support moving from 28% in one ICM poll to 34% in a later YouGov offering as a jump of 6 points for the party. ICM polls should only be compared to earlier ICM polls, and YouGov to preceding YouGov polls. Quite apart from anything else, this should give a much less volatile picture of poll movements, and make it easier to chart the long-term trends.

It would also be a mistake to read voting intention figures as directly comparable to the make up of any future Parliament. Due to the mysteries of the boundaries system, Labour requires a much smaller share of the popular vote to secure a working majority than the Tories do. Broadly speaking, the Conservatives need a lead of around 9 points in the polls to be confident of forming a majority government, and even then they are susceptible to being scuppered by a strong Lib Dem showing. The reasons for this apparent bias can be found in the over-representation of certain areas, notably Wales, the often ‘clustered’ nature of Tory support, and the greater propensity for Lib Dem and Labour voters to vote tactically against the Tories.

To complicate matters further, top-line figures fail to take account of the regional variations that often decide elections. Polls are weighted to reflect national data, when in fact it is usually a relatively small number of marginal seats that decide the eventual outcome of a general election. This is particularly pertinent in the 2010 election, where the Conservatives have invested vast sums – controversially backed by Michael Ashcroft’s donations - in campaigning in marginal seats. If this strategy were to pay off, the Tories could notionally win enough seats to form a government with a narrower lead than might be expected in the national polls. The key piece of data here is the swing in voting intention against the swing required by the challenger to unseat the incumbent. In 2010, the bulk of Labour’s key marginals are in the south east of England, whilst the Tories will also need to make in-roads into a number of target seats held by Lib Dems in the south west if they are to enjoy a healthy majority. Top-line figures therefore often fail to reflect the nuances at a regional level that can have a profound effect upon the eventual result.

It’s perhaps not surprising that the voting intention by party figures should grab the headlines, but it is also worth taking these within some kind of context. For instance, whilst voting intention figures widely predicted a Labour victory prior to the 1992 general election, John Major retained a lead over Neil Kinnock in his personal approval ratings throughout the campaign, mirroring the eventual result much more accurately. Similarly, party ratings in individual policy areas can prove a useful barometer should a single area come to dominate the election coverage, as the economy threatens to do in 2010. Whilst it is undoubtedly easier to spot these trends with the benefit of hindsight, looking beyond the top-line figures can help analysts spot anomalous figures.

Another bugbear for pollsters is the leading question; often a favourite of pressure groups with an agenda to push. Examples of such shady dealings include prompting - offering the respondent an option where it isn’t merited, and artificially shifting the weight of possible answers in order to favour a given outcome (as in ‘would you say this issue is extremely important, very important, important, not important’). Mercifully these false positives are easily spotted with a quick glance through the polling tables, but this is unlikely to stop the commissioning organisation from formulating some sensationalist headlines, which are often regurgitated without thought in the media.

Chilcot-en Gains? Have Lessons Been Learned Over Iraq?

In early 2003, as the march to war in Iraq began to accelerate, I can remember reading an excellent report in The Guardian’s G2 supplement by someone formerly in the military’s top brass, whose name now escapes me. The reason the article has stayed with me seven years on is that, at a time when the commentary in the media churned out claim and counter-claim about Iraq’s presumed stockpile of WMDs, this lone voice was urging the west to consider something that was all but ignored at the time: Iraq’s domestic politics.

Iraq was disarmed after the first Gulf War in 1991 and, the argument ran, if we are to go to war over the country’s weapons of mass destruction, the burden of proving their existence falls upon the invading powers. Granted, Iraq had failed to comply with UN weapons inspections, but one should consider the unenviable position Saddam Hussein found himself in through the late 1990s and early 2000s. His outward shows of defiance not only masked his weakening hold of power in Iraq, but were actually the only thing keeping the dictator from losing popular support, itself the only thing propping up his regime. UN incursions into Iraq not only undermined Saddam’s authority, but also placed pressure on him to sate the anti-western sentiment held by many Iraqis.

Fast-forward to 2010, and what has struck me most about the latest bout of hand-wringing at the Chilcot inquiry is the extent to which the terms of the debate are still grounded in these pre-war assumptions. Gordon Brown asserted in his evidence that Saddam ‘did not disclose, far less dismantle any of his weapons’, a claim that was not questioned by a single member of the Chilcot panel. Even where mistakes have been admitted, politicians have begged exoneration on the basis that their decisions were made in good faith. Take Tony Blair, for example, claiming that ‘What I actually said in the foreword [to the September 2002 Iraq dossier] was that I believed it beyond doubt [that Iraq possessed WMDs] and I did believe it’

Politicians, of course, are not exactly known for their heart-felt apologies. However what worries me most about the ongoing Chilcot inquiry is that I’ve seen very little evidence that lessons have been learned from the whole debacle. David Miliband, for example, claimed in his evidence yesterday that ‘It was no good the UN passing strong resolutions if it was then feeble in its follow-up.’ But how could Iraq have complied with the demands of the resolution referred to by Mili senior – Security Resolution 1441, the final request made by the UN for Iraq to disarm – if the intelligence that suggested that Iraq was ‘armed’ in the first place was at fault? If this inquiry has revealed anything, it is that the burden of proof should have fallen on the US and the UK to substantiate the claims they made in the build up to war. Judging from what I have seen, I can’t see that the present Government would react any differently if it were presented by a similar scenario in the future.