Saw this earlier.
Thought it was interesting.
Forgive my formatting as I'm on mobile.
The Upshot
THE 2016 RACE
How One 19-Year-Old Illinois Man Is Distorting National Polling Averages
505
By NATE COHN
OCTOBER 12, 2016
There is a 19-year-old black man in Illinois who has no idea of the role he is playing in this election.
He is sure he is going to vote for Donald J. Trump.
And he has been held up as proof by conservatives — including outlets like Breitbart News and The New York Post — that Mr. Trump is excelling among black voters. He has even played a modest role in shifting entire polling aggregates, like the Real Clear Politics average, toward Mr. Trump.
How? He’s a panelist on the U.S.C. Dornsife/Los Angeles Times Daybreak poll, which has emerged as the biggest polling outlier of the presidential campaign. Despite falling behind by double digits in some national surveys, Mr. Trump has generally led in the U.S.C./LAT poll. He held the lead for a full month until Wednesday, when Hillary Clinton took a nominal lead.
Our Trump-supporting friend in Illinois is a surprisingly big part of the reason. In some polls, he’s weighted as much as 30 times more than the average respondent, and as much as 300 times more than the least-weighted respondent.
Alone, he has been enough to put Mr. Trump in double digits of support among black voters. He can improve Mr. Trump’s margin by 1 point in the survey, even though he is one of around 3,000 panelists.
He is also the reason Mrs. Clinton took the lead in the U.S.C./LAT poll for the first time in a month on Wednesday. The poll includes only the last seven days of respondents, and he hasn’t taken the poll since Oct. 4. Mrs. Clinton surged once he was out of the sample for the first time in several weeks.
How has he made such a difference? And why has the poll been such an outlier? It’s because the U.S.C./LAT poll made a number of unusual decisions in designing and weighting its survey.
It’s worth noting that this analysis is possible only because the poll is extremely and admirably transparent: It has published a data set and the documentation necessary to replicate the survey.
Not all of the poll’s choices were bound to help Mr. Trump. But some were, and it all combined with some very bad luck to produce one of the most persistent outliers in recent elections.
Tiny Groups, Big Weights
Just about every survey is weighted — adjusted to match the demographic characteristics of the population, often by age, race, sex and education, among other variables.
The U.S.C./LAT poll is no exception, but it makes two unusual decisions that combine to produce an odd result.
■ It weights for very tiny groups, which results in big weights.
A typical national survey usually weights to make sure it’s representative across pretty broad categories, like the right number of men or the right number of people 18 to 29.
The U.S.C./LAT poll weights for many tiny categories: like 18-to-21-year-old men, which U.S.C./LAT estimates make up around 3.3 percent of the adult citizen population. Weighting simply for 18-to-21-year-olds would be pretty bold for a political survey; 18-to-21-year-old men is really unusual.
On its own, there’s nothing necessarily wrong with weighting for small categories like this. But it’s risky: Filling up all of these tiny categories generally requires more weighting.
A run of the U.S.C./LAT poll, for instance, might have only 15 or so 18-to-21-year-old men. But for those voters to make up 3.3 percent of the weighted sample, these 15 voters have to count as much as 86 people — an average weight of 5.7.
When you start considering the competing demands across multiple categories, it can quickly become necessary to give an astonishing amount of extra weight to particularly underrepresented voters — like 18-to-21-year-old black men.
Interactive Feature | 2016 Election Forecast: Who Will Be President? The Upshot’s presidential forecast, updated daily.
This wouldn’t be a problem with broader categories, like those 18 to 29, and there aren’t very many national polls that are weighting respondents up by more than eight or 10-fold. The extreme weights for the 19-year-old black Trump voter in Illinois are not normal.
■ It weights by past vote.
The U.S.C./LAT poll does something else that’s really unusual: It weights the sample according to how people said they voted in the 2012 election.
Its weights are such that Obama voters represent 27 percent of the sample and Romney voters represent 25 percent, reflecting the split of 51 to 47 percent among actual voters in 2012. The rest include those who stayed home or who are newly eligible to vote.
I’m not aware of any reputable public survey that weights self-reported past vote back to the actual reported results of an election.
You can read more about the U.S.C./LAT “past vote” issue in this August article, but the big problem is that people don’t report their past vote very accurately. They tend to over-report three things: voting, voting for the winner and voting for some other candidate. They underreport voting for the loser.
Donald Trump supporters in Panama City Beach, Fla., this week.
STEPHEN CROWLEY / THE NEW YORK TIMES
The same thing is true in the U.S.C./LAT poll. If the survey didn’t include a past vote weight, the past vote of its respondents would be Obama 38, Romney 30. This is a lot like national surveys that were published around the same time as the U.S.C./LAT poll, like those from NBC/WSJ or the NYT/CBS News.
By emphasizing past vote, they might significantly underweight those who claim to have voted for Mr. Obama and give much more weight to people who say they didn’t vote.
Two Key Factors
These two factors — an overweighted sample and the use of past vote — seem to explain the preponderance of the difference between the U.S.C./LAT poll and other surveys.
If the poll was weighted to a generic set of census categories like most surveys (four categories of age, five categories of education, gender and four categories of race and Hispanic origin), Mrs. Clinton would have led in every iteration of the survey except the period immediately after the Republican convention. The U.S.C./LAT poll weights for all of these demographic categories; it just weights to smaller groups.
About half of the difference is attributable to the small demographic categories that lead the 19-year-old black Trump voter in Illinois to get huge weights. The other half of the difference is because of the past vote weight.
Of the two factors, it was probably inevitable that using “past vote” would create a problem. The potential biases of weighting by past vote are pretty well established.
But the costs of the U.S.C./LAT poll’s extensive weighting were not so inevitable.
Jill Darling, the survey director at the U.S.C. Center for Economic and Social Research, noted that they had decided not to “trim” the weights (that’s when a poll prevents one person from being weighted up by more than some amount, like five or 10) because the sample would otherwise underrepresent African-American and young voters.
This makes sense. Gallup got itself into trouble for this reason in 2012: It trimmed its weights, and nonwhite voters were underrepresented.
In general, the choice in “trimming” weights is between bias and variance in the results of the poll. If you trim the weights, your sample will be biased — it might not include enough of the voters who tend to be underrepresented. If you don’t trim the weights, a few heavily weighted respondents could have the power to sway the survey. The poll might be a little noisier, and the margin of error higher (note that the margin of error on the U.S.C./LAT poll for black voters surges every time the heavily weighted young black voter enters the survey).
But the U.S.C./LAT poll is a panel — which means it recontacts the same voters over and over — and so it wound up with the worst of both worlds.
If the U.S.C./LAT poll were a normal poll, the 19-year-old from Illinois might have been in the poll only once. Most of the time, the heavily weighted young black voters would lean toward Mrs. Clinton — ensuring that the poll both had the appropriate number of black voters, and a relatively representative result.
But the U.S.C./LAT poll had terrible luck: The single most overweighted person in the survey was unrepresentative of his demographic group. The people running the poll basically got stuck at the extreme of the added variance.
By design, the U.S.C./LAT poll is stuck with the respondents it has. If it had a slightly too Republican sample from the start — and it seems it did, regardless of weighting — there was little it could do about it.
The Upshot provides news, analysis and graphics about politics, policy and...
Poll
