Guest post by Geoff Nunberg, Professor at the UC Berkeley School of Information and linguistics expert for the petitioners in the Blackhorse challenge to the Redskins trademark registration.

“The goal of the study was never to generate nationally representative estimates specifically for the Native American population.”

That’s the crucial take-away from the memo prepared by two senior researchers at the survey organization that conducted the often-cited 2004 National Annenberg Election Survey (NAES), which reported that only 9 percent of self-identified Native Americans found the name of the Washington Redskins offensive. The team’s owner Dan Snyder and its representatives have repeatedly pointed to that result to support their contention that few Native Americans are bothered by the name. But the researchers’ memo definitively disqualifies the survey as an accurate picture of Native American attitudes about the name. As the memo goes on to say, “it is not appropriate to use NAES data to study this population.”

The memo was submitted to Ken Winneg, Annenberg’s Managing Director of Survey Research by Chintan Turakhia, Sr. and Courtney Kennedy, both vice-presidents and senior researchers at Abt SRBI, the survey organization responsible for collecting the data for the 2004 NAES. Winneg asked them to respond to a letter I had sent him with some questions about the survey. As the linguistics expert for the petitioners who asked that the team’s trademark registration be cancelled, I had been puzzled by the result, which seemed inconsistent with other surveys. Among other things, it struck me that given the proportion of Native Americans who responded, the sample couldn’t have been representative of national attitudes. In the light of those problems, could the NAES survey be used as evidence of Native American attitudes about the name? I wrote to Dr. Winneg, who graciously asked Turakhia and Kennedy to prepare a response. They wrote:

…the goal of the study was never to generate nationally representative estimates specifically for the Native American population. The design and implementation of the 2004 NAES was appropriate for the main research goal of the study, which was to generate a nationally representative sample of U.S. adults. Even very large probability‐based samples, like the 2004 NAES, are not al ways effective for generating representative samples for all possible subgroups that may be of interest. Some subgroups, including Native Americans, have unique characteristics (e.g., multiple languages, unusual residential patterns) that require specialized survey designs if they are to be measured rigorously.

Native Americans are well known to be a relatively small and difficult to survey population. As noted in the memo, landline telephone penetration rates on reservations are significantly lower than they are elsewhere in the U.S. The experiences and attitudes of Native Americans living on reservations may very well be different from those living elsewhere. A survey designed specifically for Native Americans would therefore need to have a special protocol for reaching those living on reservations. Given that our goal was never to generate nationally representative estimates for Native Americans, these initiatives were not built into the design. Due to the above reasons, it is not appropriate to use NAES data to study this population.…We feel it would be more appropriate for those engaged in this discussion to consider more recent research from studies designed specifically for the Native American population.

The full text of the memo is here.

Why isn’t the survey an accurate picture of Native American attitudes?

What was the problem here? The Annenberg Public Policy Center is extremely reputable, and it may seem curious that the Center’s own pollsters would disown the significance of their results. But the circumstances of the 2004 “Redskins” question are exceptional. The item was tacked on to the survey as an afterthought by Adam Clymer, Annenberg’s political director, who thought it would make for a nice short piece and who believed that survey’s large sample would reach “a statistically meaningful number of respondents identifying themselves as Indian or Native American.” Clymer is a fine journalist but he isn’t himself a survey expert and obviously didn’t anticipate the difficulty of simply adding that question to the survey. As Turakhia and Kennedy note, a survey designed to reach a broad sample of American voters is not necessarily going to reach a representative sample of every possible subgroup, particularly when that subgroup has characteristics that make its members hard to access. That won’t have any major effect on the survey’s general results about political questions, where the opinions of Native Americans represent barely one percent of the overall sample. But it will obvious skew the results of a question aimed specifically at that group.

Why is the Annenberg Survey unreliable?

There are a lot of reasons why the NAES survey can’t be relied on as a snapshot of 2004 attitudes. A number of these were covered in a 2014 post at this site, “11 Reasons to Ignore the 10-Year-Old Annenberg Survey About the Washington Football Team’s Offensive Name.” Several stand out:

The landline problem

As Turakhia and Kennedy note, a landline survey would be certain to miss many Native Americans, particularly those who live on reservations. As late as 2013, the FCC reported that only 37 percent of Navajo residents had access to either landline or cell service. A 1995 Census brief reported that 53 percent of all American Indian households on reservations lacked telephone service; on some reservations in Arizona and New Mexico (including the largest), the figure was over 80 percent. That didn’t simply mean that the survey missed a large number of Native Americans, but that the sample it did reach would be skewed—towards self-identified Native Americans with landlines who were willing to cooperate in a survey. And as it turns out, those would tend to be people with weaker cultural and tribal ties, who disproportionately live in urban areas.

The survey was skewed

The 2000 census was the first to distinguish those who identified as “Indian only” and “Indian in combination with other ethnic identities.” Not surprisingly, the proportion of those identifying as Indian only is higher in the West (63 percent) than in the Northeast (42 percent), and among the two largest groups, far higher among those identifying as Navaho (89 percent) than those identifying as Cherokee (35 percent). It is far higher in New Mexico (86 percent) and Arizona (86 percent) than in Ohio (37 percent) or Pennsylvania (33 percent).

We can assume, then, that since the NAES survey considerably undersampled Indians on Western reservations and in rural areas, it under sampled those who identify as Indian-only against Indian plus. But the former are far more likely to have a strong Native American cultural identity, to live in close contact with other Native Americans, to have a tribal affiliation and to speak a Native language. In the nature of things, they’re also more likely to have experienced racial discrimination than someone who’s a half or a quarter Native American who lives in an Eastern suburb and whose identity isn’t evident to others—much less someone whose claim to Native American identity is based on a family tradition about a Cherokee ancestor and who has never been slightingly addressed as a redskin. (The comments threads on articles about the name are full of remarks like “I’m part Indian & proud of it & have no problem with the name ‘The Redskins’!”; they cry out for a response like “I’m proud to be one quarter white and I have no problem with ‘honky.’) As Turakhia and Kennedy note, it isn’t clear how many of those who identify as Native American plus will choose Native American as their main identification on the NAES survey. But it’s worth noting that roughly 45 percent of the growth in the self-identified Native Americans since 1960 has been among those who switched from another identity.

In short, the NAES sample wasn’t simply unrepresentative of Native Americans, but was almost certainly skewed with regard to the cultural and demographic factors that would shape attitudes about the Redskins name. As Turakhia and Kennedy note, “it would be more appropriate for those engaged in this discussion to consider more recent research from studies designed specifically for the Native American population.”

Turakhia and Kennedy say, “If generating estimates specifically for the Native American population had been a main research goal of the study, then it would be reasonable to criticize how race [was] measured.” That’s certainly true as regards the main goals of the survey, of capturing national electoral attitudes. But the methodology of a study has to be evaluated with regard to all of its goals and the use that is made of its results. On the survey’s completion in 2004, Annenberg issued a press release headed “Most Indians Say Name of Washington “Redskins” Is Acceptable, While 9 Percent Call It Offensive, Annenberg Data Show.” As Turakhia and Kennedy acknowledge, that report was not accurate as a picture of national Native American attitudes. Given the political significance that the team and its supporters have attached to that result, the inclusion of those questions was unfortunate. Not that anyone at Annenberg had the slightest intent of misrepresenting Native American attitudes—clearly the wisdom of tacking on these questions wasn’t thought through. But in retrospect, it would clearly have been better if the questions had not been included at all.