Public Policy Polling (PPP) is a long-time Democratic pollster, which last year became the official pollster for DailyKos.
PPP is prolific in its polling, with new polls released almost daily. I have linked to PPP’s polls in the past. PPP has a good knack for finding headline-grabbing topics.
But since the November elections, I have noticed an almost non-stop release of polls which are presented by PPP as signaling bad news for Republican. It is as if on November 2 Republicans were the toast of the country, but on November 3 they were just toast.
Perhaps because PPP has a tendency to put a very political spin on its polling results, and because the spin has been doom and gloom for Republicans, I have begun receiving a steady stream of e-mails complaining that PPP’s polling samples were skewed against Republicans.
I pretty much wrote off such complaints at first as the usual type of griping which happens when people don’t like poll results. Rasmussen regularly is skewered in the left-blogosphere because the left-wing blogs don’t like the results. If PPP’s polling samples and methodology are legitimate, it is not fair to criticize the results. Maybe the world did end for Republicans on November 3.
But a poll released yesterday by PPP caused me finally to dig into PPP’s sample data. In a poll which has received enormous attention and warmed the hearts of public sector union supporters, PPP found that in a “do over,” current Wisconsin Governor Scott Walker would lose to the person he beat in November, Tom Barrett.
Here is part of how PPP’s Tom Jensen described the results:
“We’ll have our full poll on the Wisconsin conflict out tomorrow but here’s the most interesting finding:if voters in the state could do it over today they’d support defeated Democratic nominee Tom Barrett over Scott Walker by a a 52-45 margin.
The difference between how folks would vote now and how they voted in November can almost all be attributed to shifts within union households. Voters who are not part of union households have barely shifted at all- they report having voted for Walker by 7 points last fall and they still say they would vote for Walker by a 4 point margin. But in households where there is a union member voters now say they’d go for Barrett by a 31 point margin, up quite a bit from the 14 point advantage they report having given him in November.”
But looking at the data behind the poll reveals that PPP used a sample skewed against Walker voters and towards union households.
The people answering the poll identified themselves as evenly split in how they voted last November, 47-47:
But the actual results in the election were in favor of Walker 52.3-46.5, as this chart from The New York Times shows.
So the sample used in the PPP poll seriously underweighted Walker voters by almost 6%.
Additionally, the PPP sample contained 32% union households:
But in Wisconsin, only 26% of households are union households, as this chart from the Milwaukee Journal Sentinel shows:
So the PPP sample overweighted union households by 6%.
Put together, the PPP voter sample used to show a huge swing since November appears to have accomplished the result by underweighting Walker voters and overweighting union households. This is particularly important since PPP claims (in the quote above) that it was a shift in union household voting patterns which was most prominent.
I posted a comment at PPP’s blog last night asking Mr. Jensen for an explanation and pointing out the very issues raised in this post. As of this writing, there has been no response. If Jensen does respond, I will post the response.
Perhaps I’m missing something here. After all, I’m not a professional pollster, I’m just someone who read the data behind the headline. I understand that polling firms sometimes adjust data when the underlying sample is skewed; but there is no indication PPP has done that.
It is not fair to criticize PPP’s results just because one doesn’t like the results. But it is fair to examine the data behind the poll, which is something I will be doing with more frequency with PPP in the upcoming election cycle.
* * * *
You can help me. If you believe that there are substantive problems with PPP polls, please e-mail me with specifics. I’m not interested in name-calling or conspiracy theories. But if you can document sampling or other errors similar to what I have done above, I would love to hear from you.
Update: Allahpundit and Ed Morrissey at HotAir also have noticed sampling problems with a CBS/NY Times poll just released purporting to show strong union support.
PPP also released at 10 this morning more complete results of its Wisconsin polling, with some more headline-grabbing conclusions — such as voters siding with the Senate Democrats — but based on the same voter sample which underweighted Walker voters and overweighted union households, and accompanied by this disclaimer which was not on the results released yesterday (emphasis mine):
“PPP surveyed 768 Wisconsin voters from February 24th to 27th. The survey’s margin of error is +/-3.5%. Other factors, such as refusal to be interviewed and weighting, may introduce additional error that is more difficult to quantify.”
Update 2:10 p.m. – Jensen has not responded, but another person at PPP (Dustin Ingalls) responded to a series of questions posed by commenters, as follows (emphasis mine):
“Is this a poll of registered or likely voters?”
Registered.“do you think there’s any reason in particular regarding the difference of reported 2008/2010 #s from actual ones?”
Yeah, different voters turning out/responding to the poll. Last year’s formula was quite simple: record GOP turnout, and a depressed Democratic base in Wisconsin meant one of the country’s largest “enthusiasm gaps.” 2012 will probably not be quite as historic a Democratic turnout as 2008 was, but much closer to 2008 than 2010. Our poll is of registered voters, but the more interested you are in politics and voting, the more likely you are to answer the phone and stay on the line for the whole poll, which mirrors their likelihood of voting. Registered voter polls for presidential elections, which have wider turnout than midterms, are closer barometers of the turnout than registered voter polls for a midterm, which see a much narrower and less predictable set of voters turn out, and tend to lean GOP anyway, particularly last year.
“Sample appears seriously skewed. 47% say voted for Walker, but reality of election result was 52.3% so seriously underweight Walker voters. Also, sample is 32% from union households, but only 26% in Wisconsin from union households, so overweight union households.”
See what I said above. A lot of potential Barrett voters who will vote in 2012 (or would vote now) stayed home last fall and have been even further galvanized by this fiasco. Also, 26% and 32% aren’t a whole lot different, but union voters are a pretty well organized bunch. They turn out at a higher rate than some non-union voters, meaning they’ll make up a larger share of the electorate than their share of the overall population.
“Did you also poll the Republicans vs Obama and/or the Republican primary? Will we have those reslts later today, tomorrow or when?”
Dude, do you understand the concept of patience? We release one set of results each day during the week. That’s been our M.O. for a long time.
March 1, 2011 1:11 PM
The 26% figure comes from 2010? Of course it’d be higher now, since union members have been stirred to action. It’d probably even be higher than in 2008 if an election were held now. If unions and the Democratic Party in the state can keep those workers angry and organized, that anger could probably cause an even bluer election in 2012 than 2008. If it subsides, you’re still looking at an election closer in turnout to 2008 than 2010. Do Republicans think 2010 was the norm and that voters shifted to the right?
March 1, 2011 1:22 PM
Seems like a very weak response to me, as it attempts to excuse a clear skew in the sample by speculating that there would be higher turnout numbers only on one side if the election were held how. Well, maybe there would be higher turnout numbers on both sides. There is no evidence given to support such speculation. I hope that this does not represent the level of PPP analysis.
——————————————–
Follow me on Twitter, Facebook, and YouTube
Visit the Legal Insurrection Shop on CafePress!
CLICK HERE FOR FULL VERSION OF THIS STORY
