When it comes to television ratings, it would be nice to leverage a next-generation set-top box (STB) to monitor exactly what everyone is watching, all the time. But technology hasn’t caught up with the wish-list, and ratings measurement is still often conducted using self-reporting via diaries and phone surveys.

The actual sample sizes used for determining ratings tends to be fairly small too, so statistical accuracy is the key to effectiveness. But whether the samples are actually representative of the viewing public remains a question. “No one has ever asked me to write down what I watch,” you may hear someone say. And anecdotally, how many people does the average person know who have been tapped to provide TV viewing information to Nielsen?

The Council for Research Excellence (CRE) just completed a two-year project to evaluate sample quality for ratings measurement, and found that those that contribute to the measurement database are not necessarily representative of the U.S. demographic makeup. 

For instance, the study discovered that respondents in the study sample tended to be older. In Dallas, 54.7 percent of sample participants (or "in-tabs") were over 50, compared to just 28.2 percent for heads of households within the 35-54-year-old age range, and just 17.2 percent of those in the under-35 age range. Non-participants ("non-intabs") outnumbered participants in the 35-to-54 and under-35 categories. The patterns were similar in the smaller markets. Similarly, ethnic representation didn’t match the greater population’s demographics.

"While these latest findings may be intuitive, even unsurprising, it's important that we now have hard information,” said Ceril Shagrin, executive vice president of Univision Communications, who serves as chair of the CRE as well as its Sample Quality Committee. "Both of our studies, for example, found that samples were missing young people and Hispanics; our new study shows the differentials are not as wide, but they still constitute a significant gap.” 

In addition to the differences between in-tab and non-in-tab homes, the study uncovered substantial information about household consumer-electronics ownership. For instance, 38.2 percent of all the households surveyed did not have a landline phone and had at least one cell phone. Also, the portion of adult heads-of-households aged 25-to-29 who use only a cell phone has grown from 46 percent in June 2009 to 62 percent in December 2012.

A 27-household sample of non-TV homes found 63 percent had a computer with high-speed Internet and 41 percent use the Internet on a cell phone; 38 percent viewed TV on a desktop PC, laptop or tablet; and of the 61 percent who still watched television on a TV set, nearly half watched in someone else's home, followed distantly by bars and restaurants.

“We needed to understand whether deeper information about media-related equipment ownership can be obtained from diary samples,” Shagrin said. “And, even though we had only a very small sample, we now have a better idea of how much 'TV program' viewing is done – and on what devices – in what are currently defined as 'non-TV homes.' This helped confirm for us that people who had not previously been measured view TV differently – and that it's important that this viewing be measured. We need to reach harder-to-reach households."

Increased movement from phone-based sampling to address-based sampling has improved television audience-sample frames, the study found. However, as evidenced by differential response rates in sampled homes, non-response bias cannot be completely resolved by weighting measures and remains a persistent challenge.

"Nielsen's address-based sampling has reduced response bias but there remains a significant difference between metered and diary samples," said Shagrin. "An improved sample frame improves representation of younger, cell phone-only homes -- which are growing rapidly -- but more needs to be done. Measures such as enlarging the sample size or weighting the results don't adequately resolve the issue; you can weight demographically, but we've learned more clearly that you can't sufficiently weight for non-response."

The three television markets included in the study were Dallas-Fort Worth, the fifth-largest TV market and a Nielsen local people-meter (LPM) market; Albuquerque-Santa Fe, the 47th-largest TV market and a standard Nielsen-meter market; and Paducah, Kentucky-Cape Girardeau, Missouri-Harrisburg, Illinois -- Nielsen market #81 and a diary market. The markets were selected due to varying measurement methodology, varying market size, and divergent characteristics, such as geographical coverage, ethnic make-up, number of over-the-air households and penetration levels of electronic devices.