I have no implicit pro- or anti-Nate Silver bias.
He was a journolist and in 2008 got some secret help with his predictions from the Obama campaign, but since joining The NY Times I think he has raised his game.
I find the whole focus on Silver and his presidential election “model” to be particularly annoying, in part because there seems to be a lack of transparency. Is the methodology and how it is used to come up with specific percentages fully disclosed?
Whatever, Silver-mania (both pro and con) seems to have invaded way too many bodies.
I was in an SUV last Wednesday night in Austin with a group of people at a conference heading to dinner. A woman in the car got a beep on her phone, and she exclaimed words to the effect: “I just got an alert, Nate Silver has Obama’s chances of winning at 72.3 percent.” (I may be off slightly on the percentage, but that’s pretty close.)
At that moment, if I did not know it before, I understood that this Nate Silver thing had gone way too far.
I’m not a numbers guy, never claimed to be. My primary strength when it comes to numbers has been to reduce complicated damages calculations to an explanation so simple that even someone as statistically illiterate as me can understand it. If I can understand the numbers, anyone can.
But there’s something that doesn’t require you be a numbers guy or gal which bothers me about Silver-mania.
Silver cannot be wrong because his model, whatever it is, merely puts a likelihood of success on the election. Silver’s model could predict a 75% chance of an Obama win, but if Romney wins the model was not wrong because his model allowed for a 25% chance of a Romney win.
If Silver cannot be wrong, how can he be right? Heads he wins, tails you lose. Calling all philosophy majors!
I’ll leave you with this. My first experience with Silver-mania was in his pre-NY Times days, when he was covering the Coakley-Brown race. According to Wikipedia, Silver’s 538 model was pure genius:
FiveThirtyEight writers Schaller, Gelman, and Silver also gave extensive coverage to the January 19, 2010 Massachusetts special election to the U.S. Senate. The “538 model” once again aggregated the disparate polls to correctly predict that the Republican Scott Brown would win.[40]
The source in the footnote was Silver’s January 18, 2010 blog entry, 538 Model Posits Brown as 3:1 Favorite posted at 5:26 p.m. the day before the election.
Well, yeah, duh, about 14 hours before the polls opened Silver joined the rest of the world in predicting a Brown win, but he was consistently predicting doom for Brown and was the last person to jump on board.
Here’s the pertinent chronology from my post in February 2010, Beware Statisticians Bearing Advice:
Nate Silver, January 5, 2010, Wicked Awesome Thoughts on Massachusetts Special Election:
Rasmussen is supposed to have a poll out tomorrow (Tuesday) on the Massachusetts Senate Special Election, which will take place on the 19th. There’s been some speculation, mostly from Republican blogs but also from some Democratic analysts, that the Republican candidate, Scott Brown, might have a chance, which would potentially wreck the Democrats’ chances to pass health care reform.
I’ll be curious to see what Rasmussen and the other pollsters (PPP? Suffolk?) have to say, and bears remembering that special elections are highly unpredictable affairs. But I’d be somewhat surprised if the election turns out to be especially competitive.
Nate Silver, January 14, 2010, OK, It’s a Toss-Up:
Earlier today I tweeted about how there wasn’t enough evidence to describe the Massachusetts special election as a “toss-up”, as some other forecasters have done, based on the information available to us at that time.
Well, now there’s some new evidence. And it isn’t good for Martha Coakley.In particular, the evidence is a Suffolk University poll that shows the Republican, Scott Brown, ahead by 4 points, 50-46.
Nate Silver, January 17, 2010, 538 Still Rates Massachusetts as Toss-up:
Please don’t be too enthralled with/scared by the specific numbers below. They’re based a number of assumptions which may not be valid. However, I agree with the characterization that the model comes to, which is that the Massachusetts special election should continue to be regarded as a toss-up.
Nate Silver, January 19, 2010, Defying Odds, Republican Brown Becomes Next Senator from Massachusetts:
In an outcome that would have been unthinkable just weeks ago, Republican Scott Brown has become the Senator-elect from Massachusetts.
Silver was as wrong as anyone could be in the Brown-Coakley race, yet came out a hero because his model predicted a 75% chance Brown would win late in the day before the election.
I neither a Nate Silver lover nor hater be.
But when someone has a model which cannot be proven right or wrong, I’m not buying whatever it is he’s selling.
DONATE
Donations tax deductible
to the full extent allowed by law.
Comments
Along with Nate Silver, I cannot understand the fascination with Intrade. Apparently all of the punters on Intrade bet on the advice of Nate Silver.
Prediction markets have had some pretty good runs in the past.
But, they trade too thinly, and anyone who wants to steer their results could with just a few thousand dollars.
I will say this, InTrade’s electoral college map was, at the end of the last week, far more “even” than the bare-bones overall percentages.
The map was supposedly made up of some battleground state bets on Intrade, instead of just winner-takes-all bets.
Still, I have said it before and will say it again and again: Each side can fear X or celebrate Y, but this is going to be a hard fought campaign all the way through election day. That does not mean that the popular vote spread will be razor thin or that the electoral college will be razor thin, but no one should get too complacent or too discouraged before the last polls close. None of the battleground states are completely out of reach of a blitzkreig ground game by either side (or October Surprises or election chicanery/fraud).
People need to realize that Intrade is quite accurate on many matters, but politics is not one of them. There is so much emotion and activism (buying shares, e.g.) in politics it will probably never be well represented by Intrade. But on most matters where buyers have an interest in honest earnings, it is accurate. With politics many buyers may have little interest in earnings.
Silver’s an idiot.
William, I’ve got you linked up: ‘Nate Silver Fast on His Way to One-Term Celebrity’.
Intrade is a very thin market. At $7 a share, someone could easily manipulate this market for propaganda purposes without risking very much money.
RickCaird,
Gave you a thumb’s up.
And, to everyone who might not have caught my second or third paragraph above, please see my post below.
Intrade has a few problems. It is relatively small, so a big infusion of cash can influence it. But the bigger issue is it lags. The stock market is no different, it moves in increments and jumps based on news. Right now conventional wisdom is Obama will win. So he is at 62%. Obama was over 80%. Pretty soon Obama will be lower yet. That does not mean Intrade is useless, but it a lagging indicator.
Polls, if done right and uniformly, polls can show trends. The trends right now are moving in Mitt Romney’s favor.
Already onto him.
He is on the “Enemies List” that I am working on. Am up to 600 names.
Gilbert & Sullivan were right:
As some day it may happen that a victim must be found,
I’ve got a little list–I’ve got a little list
Of society offenders who might well be underground,
And who never would be missed–who never would be missed!
It’s sort of like trying to figure out how good a weatherman is at predicting the rain — if you only get to check his predictions once every couple of years.
When the guy arbitrarily “weights” the value of each poll and gives greater weight to a week-old poll from the same pollster (which happened to be better for Obama) and refuses to explain his weighting logic or process, I have to assume he is giving the greater weight to the results he likes.
You can call it whatever you wish, but it is NOT science.
My prediction is that all polls this week will start to shift massively towards Romney, as all the D+7/D+8 polls realize they are going to lose ALL credibility if they are that far off! They will of course claim some other reason for the last-minute shift.
Estragon and clintack,
You are both correct.
I know some pollsters and I know just enough about statistics to be “dangerous.’
Obviously, conducting the poll is Step 1, i.e., how the questions are asked, phrased, and in what order, as well as the tone/tenor of the person doing the asking, and making sure that a fairly good sample is, well, “sampled.”
Then, Step 2 is analyzing the data. Was there some sort of framing bias, sampling bias (say, because of time of day, cell phones vs. land lines, caller ID, geographic clustering, etc.)
Then, comes the Art of Polling: Adjusting the data to correct for all of those things, which is all the more difficult in the particular genre of political polling, because 2006, 2008, and 2010 were “volatile” elections in which the turnouts varied from the previous two-year elections of 2004, 2006 and 2008.
It is a tough time for pollsters. They are in uncharted territory.
My work involves (amongst other things) a particular type of asset valuation in which the volume of prior transactions is way down from before 2008 and which involves a lot of other “noise.”
In short, pollsters are waterskiing blindfolded. After the election, someone will emerge as being more accurate than most everyone else, and, their method(s) may even be celebrated and studied. But, before that Monday Morning Quarterbacking takes place, I can assure you that they are all suffering from “heartburn” right now.
Remember how celebrated Zogby was for a few years? Do you remember when Zogby started to drift? Do you know what event preceded his drift and what it might have to do with him (or his brother)?
I guess a good analogy would be money managers who use active management. Some of them beat the market for quite a few years, but none of them have “beaten” the market consistently. Warren Buffett is not, per se, a money manager, but you could say that he comes the closest. Would that it were that Mr. Buffett would keep his mouth shut about politics. Sadly, he has not. Some might wonder whether his pronouncements are something other than purely objective.
I guess a good analogy would be money managers who use active management. Some of them beat the market for quite a few years, but none of them have “beaten” the market consistently.
Renaissance?
Let me know if I am wrong. I had never heard of this guy before.
My understanding is that Silver was a DailyKos guy, must not be that smart. Apparently he had impressive accuracy in the ’08 election. Then he was flat wrong in the ’10 midterms. He said there was a 70% chance that Repubs would be held below some number of gains (60 seats?) and was wrong (gained 63?).
Now he is 70% sure (that number again!) that Obama will win.
1) he’s not always right
2) he doesn’t have a long track record
What is the fuss about?
Also, Krugman approves of the guy. Another warning sign!
http://dailycaller.com/2012/10/29/protecting-nate-silver/
Here’s my prediction: After this election you will not be hearing much about Nate Silver.
The more I think about 538, the more troubling I find Silver’s use of a percentage to express the supposed state of a race. It seems to me that if all NS is doing is analyzing poll data (and not, for example, predicting the performance of GOTV efforts), then he ought to be able to state that Candidate X will get Y percentage of the vote, not that he has a Y percent chance of winning. If the candidate in fact is due to get say 52% of the vote, then he has a 100% chance of winning the election. The fact that Silver expresses his conclusions in terms of fractional probabilities simply implies that his analysis of the data doesn’t yield a definitive answer as to which candidate is actually ahead. Thus, when he gives BHO a 75% chance of beating Romney, he’s really saying he THINKS Obama is ahead, but he’s really not sure. Add to this uncertainty the fact that the model really CAN’T take into account either future movement of voters toward or away from each candidate OR turnout/GOTV and I’m left wondering how much value NS is adding to the debate.
Exactly right. The guy is fundamentally ignorant about stats. He understands not one whit if he talks about percentages for a single event. Probabilities applies multiple trials. There will be only one election.
If a roll a pair of dice ONCE, what is the probability that I will roll snake eyes? 1 out of 36? No! It’s either 0% or 100%. Either I do or a don’t.
I read 538 a few times in 08. I would never read the NYT.
This year i go to battlegroundwatch, which has been agitator free until yesterday.
Sorry you had to be stuck in Austin with a “Dim”ocrat, Professor. Austin would be a beautiful city if it weren’t for the libs that have taken it over.
So he has a history of being “wrong” (if he says 51% he is picking, as I see it), but he says what people want to hear.
As with a religious cult, if you dress things up and make people feel you have some special knowledge, they might follow you. Especially when that special insight sets them apart from those that think for themselves, and in an area where there is emotion and doubt.
Stock market gurus are the same breed … don’t outperform the S&P, but can talk your ear off about why you should buy their hot tips. And they will give you the times they are right as evidence.
Romney wins with 307 electoral votes. I’m 100% sure. 😉 Now see, if that comes true, I will be the next guru, and Silver will slither away, as I mock his foolishness.
My post above might not have been as well written as it should have been, not that anyone has called me out.
Of course, like a responsible person, I put the blame entirely on myself.
Last Friday, I was surfing around and noticed something. InTrade has overall odds at, say 65-35, based upon WTA (winner take all) bets by Intrade “customers.” The 65-35 being in President Obama’s favor, but wait…
It appears that InTrade has some markets that are devoted to individual battleground states, and, when InTrade creates its own electoral college tally, based upon whatever proprietary method they use to weight overall winner-take-all bets and individual battleground state bets, the InTrade electoral college prediction last Friday (+/-) was for Romney.
BUT, none of this is conclusive, well, of anything.
Bottom line: My guess is that this election is going to be hard fought and winable by either side through the close of the West Coast polls. Typically, a spread of 4 percent in the overall national vote total has never been offset by the results of the electoral college. That long-term trend may hold again. No reason to bet that it would not. It could be, however, that this election is the aberration in which the overall popular vote does one thing and the electoral college does another.
That history-making possibility aside, the practical analysis is that a surge or some funny stuff in just one or two states could tip the electoral college.
And, none of that has anything to do with “honest to goodness” external shocks to the election, such as international affairs, or internal shocks such as a gaffe or a slip up or a loose-cannon ally/surrogate here or there.
Bottom Line: Get ten of your friends to the polls, and, politely, get them to get another ten of their friends to the polls, and, if possible, without alienating people, ten more of each of their friends to the polls.
The electoral college is designed, in part, to magnify the outcome of the popular vote. It is very likely to come out somewhere between 225 and 313. And, it is 100% likely to come out somewhere between 538 and 0, if you get my drift.
I may be a numbers nerd to a degree. Assigning probabilities presupposes that factors and conditions are not perfectly known. Some are weighted (given lower or higher priority) and some are not included simply because they are not known. When someone like Silver assigns probabilities that don’t ‘smell’ right in view of nearly all polling, Democrat favored and the rest, it is easy to decide there are inaccurate weightings of factors and perhaps unwarranted eliminations. The fact that Silver shows perfectly no signs of wanting to be fair in my experience from a brief sampling of his NYT ‘work”, it is not unreasonable to conclude his assignment of probabilities follows some undefined methodology (so he can defend it post election) but is far, far from serious and rigorous.
It was interesting to note the reference used by him on Jan. 14, 2010 . . . “a Suffolk University poll that shows the Republican, Scott Brown, ahead by 4 points, 50-46.”
In 2010, Suffolk University called the MA. special election correctly when almost all the others were calling it for the Dems. They are also the same Suffolk University pollsters who were called out for their alleged “Republican bias” when they called Florida, North Carolina and Virginia for Romney two weeks ago. It would be nice if they are proven to have made the correct call once again this year, wouldn’t it ?
Btw, Nate Silver’s biggest value to the Dems might be the degree to which Dems can point to his “70% chance of winning” as PROOF POSITIVE that their candidate is GOING TO WIN ! You can see their indoctrination into the “Silver is right” brainwashed club in their comments at the NYT, Kos, Salon, etc. every day.
I’m not a numbers gal either, but… when Silver was exercising his clairvoyance, not a single poll had Romney up in Ohio, so there was a chance that R can win the popular vote, but lose the electoral college. Now Rasmussen has him up in Ohio.
If I were to watch Tiger Woods attempt a 10 foot putt, when the ball passes the 9-foot mark, I could predict with remarkable accuracy whether or not it was going in the hole or not. And knowing that Tiger Woods hits X% of his 10-foot putts, I would of course start by predicting his chances at X% and then hone it down as it got closer. For all his wonderful modeling, that’s really what his 2008 *success* was all about.
In real life it it is all over for Tiger.
Ellin traded up with her own island & a billionaire.
I’m sure Silver is a smart guy, so he has to know something is wrong with his model. There are too many data points indicating a lead for Romney (polling of independent voters, party ID, ‘enthusiasm’ levels, incumbent approval rating, etc). What’s amazing is they’re all saying the same thing – about a 4-6 point lead for Romney. What has to really be scary for Silver et al is when his polls are corrected for voter ID in 2012 (as opposed to the previous election) they’re showing Romney with a solid lead (commensurate with a national lead in the mid single-digits).
My guess is he flips everything next weekend.
I have two takeaways from the Silver tweetmageddon over the past day.
One, Nate should be figuring out that an endorsement (and subsequent retweeting by Silver) by Krugman is more trouble than it’s worth.
Two, like the old witch trials, Silver seems to straddle logical thought. I expect pollsters and modelers to be wrong some of the time (think of how often Bradshaw, Long and Johnson are wrong about football results). But the whole if she floats, she’s a witch! And if she sinks, she’s a good Christian gal take on the polls irks me.
Actually, Professor, there is one way Silver’s model can be shown to be…well, not “wrong,” precisely, but massively in error. It’s like this:
Silver’s prognostications are expressed in precise percentages, but he leaves a lot of error room in his forecasts. Currently, he has the electoral vote forecast at 294.6 to 243.4, advantage President Obama. But both those numbers are ±61 (at the moment). So Obama’s range is from roughly 234 to 356, and Romney’s is from roughly 182 to 304. Similarly, his popular vote forecast (currently 50.3-48.8, Obama) has a current ± of 2.4, putting Obama’s range from 47.9 to 52.7 and Romney’s from 46.4 to 51.2.
So how do we prove him wrong? Simple. Make sure Romney gets more than 51.2% of the vote, and more than 304 electoral votes. Or whatever the equivalent features would be on Election Night. The point is…put the race outside Silver’s margin of error.
Oh, I know he’ll claim a 95% confidence interval…that 1 time in 20 a model should be broken. And obviously it’s easier said than done — kind of like belling the cat. But results like that would put a dent in his credibility, and we should all vote and get other people out to vote to put those numbers as close as we can. Unfortunately, I have a feeling that last-minute “adjustments” to the polls he aggregates may leave his results defensible after all. Still, we can always hope.
We conservatives rely on the “gold” standard in all things.
So leave it to the corrupt hacks at the NY Times to rely on a “silver” standard.
(Hasn’t the Times gone broke yet?)
Analyzing poll results is akin to predicting the direction the circus parade is headed by sweeping up the dung in the rear. This year I found Twitter a great way to get ahead of the polls. The tweets after a debate or other event really predicted a lot of the movement in the polls a few days later.
I’d far prefer that someone quantify their prediction than just make a call. In this case, in particular, the model has been described and so it explicitly describes uncertainty…. something inherent in all predictions.
polls have influence. many polls are just propaganda.
Soon many polls will turn to Romney. They have to (assuming that really IS the right call) or they lose ALL credibility for the next election. (some real pollster brought this up on FOX once)
But if they predict incorrectly early on, until the last week, they were never really predictive. Predictions have to foresee early, not tell us they were right all along, but voters changed in the last week.
Bu this election is different. Can we fire a black man, despite the racist overtones that action carries, thanks to decades of affirmative action and Al Sharptons? And have the $billions$ Obama has tossed toward his friends so corrupted the system, that an informed and concerned America can’t overcome the political mercenaries? And are SO many SO dependent on the government teat that they will never voluntarily vote for fiscal discipline?
Who?
You can’t take Mr. Silver to the bank. He’d been turned away for not being credit worthy.
I predict that people will massively vote for competence and trustworthiness – Mitt Romney – and not for the manufactured likeability of Obama.
Attention Dems—Nate Silver says that B.O. has this thing “wrapped up”. No need for you to vote more than once.
In Austin were we? First time? What did you think? As a rare member of that species “native austinite” I could given a few tips to keep away from the overpriced, overhyped, over fantazed parts of Austin. Anyway next time let me know. I’ll leave with a country song:
I’ve seen miles and miles of bike lanes,
In my car as I go by,
I’ve seen miles and miles of bike lanes,
They’ll be empty til I die
The definition of “low,” someone who raised his game after joining the New York Times.
Perhaps if Republicans were not so doubtful of science you could have your own site with a statistical examination of election predictions in a Republican leaning manner. It could use only Fox News polls so would then show a 100% chance of a Romney victory and you could all sleep better. Also would show a 100% chance that Obama is of Kenyan birth and is an avowed Muslim.
Shut up, Einstein. Shut up, Voltaire. I want to hear what Jay Elbee has to say about Republicans, science, Fox News, MSNBC, and the heavily-promoted Laurence O-Donnell/Tagg Rmoney bout.
Is this the Jay Elbee who works for Sen. Jon Tester?
Sure, it’s reckless to rely on a single indicator for something as complicated as a close, volatile national election, but…
Would this thread have all the criticism of polls, pundits, and prediction markets if they were giving the edge to Romney?
It seems to me that to really test Silver’s election eve predictions, you would want to look at a large number of them and see if the favored/non-favored split breaks down the way he says. In other words, if he says there is a 75% chance that Will Ferrell wins, then for Silver to be right, over time the Ferrells of the world should win 75% of the time and the Galafinackises should win 25%.
Of course, even if you could set that up, and even assuming his model doesn’t change, that only tests his election eve predictions. His pre-election predictions are untestable.
There are really two problems with Silver’s model. Let’s stipulate for the moment that his weighting of various polls based on several factors (track record, number polled, how recent, and I’ll even let him slide on his fudge factor, which is whether the result of the poll seems in line with the way he or his system perceive how the race is going in a particular state at the moment the poll was taken). Even so, there are still two problems.
The first is that he had access to the Obama internal polls last time. That helped his accuracy. Obama’s people think they’re leading all the battleground states according to their internal polling. What if their turnout model is wrong? What if they’re lying to Silver this time?
The second is larger and more difficult for Silver’s model to handle. What if the media polls his model relies on, or a great majority of them, have bad turnout models this time? For example, they could be oversampling early voters (and they are) in states where their poll says Obama is way out in front with early voters (and they are, especially in Ohio). What if a poll organization he says was inaccurate, or has a “house bias” in past elections, is deliberately adjusting its model to try to eliminate that error?
Let’s say a pollster is polling party ID and decides that although it was D+7 for the 2008 election, it’s now R+2. And maybe this pollster says, hmmm, last time, most of my state polls understated Obama’s support. Maybe I’ll adjust my turnout model to D+4, even though my own party ID poll says R+2. Maybe I’ll be closer to the mark this time. Let’s call this hypothetical pollster, oh, I dunno…Masrussen.
Why, then, Mr. Silver’s model wouldn’t take that into account, would it? It would be stuck with the same weighting, because it doesn’t know that Masrussen is adjusting (and probably over-adjusting) to try to fix it’s problems from 2008.
And then Mr. Silver’s model would give less weight to a pollster who probably is closer to the actual turnout than those who are forecasting the same turnout as last time (and in some cases, a bigger D advantage than last time, especially in Ohio).
And then Mr. Silver’s model is in a position to fall flat on its face a week from today.
Shorter version of the above 7 paragraphs: Garbage In = Garbage Out.
Nate silver’s projections consistently are wrong until the day before the election and suddenly they are then right.
Sorry, but thats just cheerleading in disguise until the final moment.
Move along
The way to measure Nate Silver (and the other quants) isn’t the top-line number: it’s his projected EV count and state-wins. If he gets 48 or 49 states right (or, uh, 50) then he’s accurate. If he lands, say 47 or below then he’s probably in big trouble for 2016 in terms of credibility.
You can do this for other sites as well. I recommend Election Projection for quants-from-the-right (the guy also does his version of unskewing which is a lot better than the general unskewing and comes with its own testable hypothesis).
http://www.electionprojection.com/index.php
As someone who’s day job is in big data (but: not a quant) I do respect this kind of analysis so I generally like what the math guys are doing, even if I don’t always agree with the outcomes–but just keep in mind that what Silver is selling is NOT a 70% chance of victory–that’s just the top-line-people-demand it. It’s the individual state counts.
Over 10 or so swing states he “should” hit his numbers more often than not so he’s reasonably testable there.
This thread is priceless, given what actually happened. Silver got 50 out of 50 states correct, and also came quite close on many vote share projections.
The average of polls predicted 49 of 50 states. His special sauce wasn’t so special