This is the first in a series GUEST POSTS by Matthew Knee, a Ph.D. candidate at Yale University specializing in campaigns and elections, ethnic voting patterns, public opinion, and quantitative and experimental approaches to political science.
____________________
Analyzing polls with only what polling companies release is a tricky business. Near-ideal poll analysis requires a database of actual, person-by-person responses, expensive software, and advanced mathematics. Ideal poll analysis requires actually being the pollster and having an overstuffed budget. However, there are a number of rules, tips, and tricks that anyone – with a bit of logic and a calculator – can use to draw meaningful conclusions from flawed polls and incomplete information.
I will be addressing these issues in three stages. In the first section, I will talk a bit about how people answer polling questions. In the second, I will discuss samples and biases. In the third, I will discuss techniques for evaluating the seriousness of bias.
All-purpose disclaimer: This series will include approximations and simplifications. It is for understanding media polls, not for writing articles for scholarly journals. It is also not exhaustive. The list of specific problems that can arise, especially in poll wording, is, obviously, enormously long.
How People Answer Polls
1. Most People Probably Don’t Think About Politics Like You Do
Most people do not think like bloggers and blog readers. We tend to have more set views on issues and are generally less susceptible (but not immune) to manipulation by bad polls or information. Most people, however, do not have a big chart in their heads listing most significant issues and a set view on each. They probably do have set answers to questions such as “which presidential candidate will you vote for next Tuesday,” but are more likely to provide inconsistent answers about say, cap and trade or even abortion.
Polling is not data retrieval. Rather, it is a process of activating associations and considerations to produce a response. People do have opinions and emotions about particular figures or values, but often do not have consistent views on specific policies. This means that, frustratingly, there is no one correct answer to what certain people think about certain topics – only how answering specific questions about them illuminates the complexities and contradictions within.
This view of polling, pioneered by UCLA’s John Zaller and others, explains the importance of wording and ordering polls correctly.
2. Question Wording And Order Can Significantly Impact Responses Through Priming and Framing
There are many ways questionnaires can be manipulated. Priming is the act of reminding a respondent of an association you want them to consider. For instance, using the phrase “Bush tax cuts” rather than “2005 tax cuts” will likely trigger whatever feelings a respondent has about Bush. An example of priming in popular culture is the beginning of South Park’s “With Apologies to Jesse Jackson” episode. The episode begins with a character about to solve a Wheel of Fortune puzzle for which the N-word is a possible (but incorrect) answer. While he thinks, the African-American cameraman peeks out from behind the camera, filling the screen with an African-American face and priming the audience to, as the character is about to, conclude that the N-word is the answer to the puzzle. (A better implementation would have shown the face before the puzzle)
Framing is the act of telling respondents how to contextualize a question. In one classic framing experiment, researchers found significant differences in college students’ willingness to allow a Klan rally on their campus based on whether it was presented as a free speech issue or a public safety issue.
These effects can carry over into later questions. For instance, people respond differently to questions on abortion when they are preceded by questions about women’s rights than when they are preceded by questions about religion.
Similarly, question order can force respondents to commit to certain positions at certain times, potentially affecting later responses by those who do not want to look like hypocrites. A Cold War era study found that 37% of respondents were willing to allow Soviet journalists in the United States, but 73% gave that answer when first asked if American journalists should be allowed in Russia.
3. Many Wording Effects Relate To Talking In Terms Of Gains Or Losses
There are a number of consistent patterns in the effects of wording. One important pattern is the difference between gains or positives and losses or negatives. People prefer beef that is 80% lean to beef that is 20% fat, or medical treatments that will save 80% to those that would let 20% die.
People are also more reluctant to take things away than to not begin giving them in the first place. People often don’t like being the bad guy. Thus, asking about taking away collective bargaining rights is in some ways a biased wording (and not just because of the loaded word “rights”) compared to asking whether or not public employees should negotiate their salaries. On the other hand, there is a real challenge to producing a media narrative that words the situation any other way.
4. There Ain’t No Such Thing As A Free Lunch – But As Long As It’s On The Menu, People Will Order It
When not forced to prioritize, people tend to prefer cutting taxes and increasing or retaining individual spending items. Combined with the inclination not to assent to negative proposals, this will lead to opposing specific cuts to government even while at the same time supporting smaller government in theory.
This also means that the tough decisions required to rein in runaway spending will usually be unpopular, at least without political cover from a severe crisis paired with bipartisan consensus or extremely unpopular scapegoats. The question facing Republicans is not necessarily “How can we balance the budget in a way that does not cost public support,” but “Given that the Democrats are unwilling to forgo political advantage for the sake of the country, how can we save our fiscal future at the lowest political cost.”
5. Options Provided Sometimes Matter
Pollsters usually provide options for respondents’ answers to each question. Sometimes the manner in which the policy space is divided can influence results. For instance, the recent NYT/CBS poll on public sector unions asks if people prefer balancing the budget by raising taxes, cutting public employee benefits, cutting roads, or cutting education. The pollsters note that a plurality prefer to raise taxes. In dividing spending cuts into multiple options, while only having one tax increase option, the poll creates the illusion that more people back tax increases than spending cuts, when in fact more people opted for the latter.
* * * *
I hope you enjoyed the first part of my polling primer. Come back tomorrow for a discussion of sampling issues.
——————————————–
Follow me on Twitter, Facebook, and YouTube
Visit the Legal Insurrection Shop on CafePress!
CLICK HERE FOR FULL VERSION OF THIS STORY