Watching You Watching TV

In 1950, 300 Chicagoans installed a new box that allowed them to order movies directly onto their televisions by calling the Zenith Radio Corporation. Phonevision subscribers paid $1 – worth about $10 today – for the device to unscramble offerings such as “Pride of the Yankees” or “Bambi.” This early experimental service not only pioneered video-on-demand, it marked the first time that a content provider knew exactly what an individual household was watching.

Today, many of our video services — whether delivered via Smart TVs (which are essentially televisions linked to the Internet), cable and satellite providers, or through on-demand streaming devices such as Apple TV — can know and track what we watch over time. This ability is turning all televisions – as well as computers and smart phones — into digital-age megaphones announcing our viewing habits every second.

Most people have learned that advertisers track their activity on the Internet. If we search for a pair of shoes or a vacation destination, related messages follow us for days. Yet few consumers realize that their televisions and related accessories know — and share — a great deal of data about them in increasingly sophisticated ways.

“It’s like a Trojan horse. I bring this TV into my house and it literally is keeping track of everything I am watching,” says Bob Meyers, a New York-based executive who has previously served as president of media at Playboy, CNBC’s senior vice president of primetime programming, and head of the research and analytics subsidiary of TiVo, TRA.

“One hundred percent of people I have ever spoken to about this issue have gotten weirded out when I talk about what is capable, even though I say ‘Oh, it is all unidentifiable because your personal information is not being recorded, it is just your viewing patterns,” Meyers said. “And they look at you like – ‘you expect me to believe that?’”

The situation has become concerning enough that the Federal Trade Commission is planning a conference on the topic for later this year. “The golden age of television has arrived with the golden age of television tracking,” the agency noted in its description of the meeting, scheduled for December.

For the most part, TV-related businesses — like companies advertising online — say they are simply grappling with how best to harness the power of big data without upsetting the customers whose information they seek.  “A lot of us are learning on the fly,” says Adam Lowy, general manager of advanced television at DISH Network. “It is a different world than it was five years ago.”

For most of television’s history broadcasters had no idea what programs any specific individual might be watching. They sent signals over the air which home receivers converted into television pictures. In their early years, cable companies also transmitted on a one-way path.

The initial Phonevision experiment in 1950 lasted only 90 days and subsequent efforts by the company never really caught on. In any case, they only knew when someone ordered a pay-per-view movie, not when the household was watching “The Lone Ranger” on regular TV.

In the 1970s, several companies sought to capitalize on two-way television. In 1977 in Columbus, Ohio, Warner Communications rolled out the first U.S. interactive television system , with 30 channels, including ten pay-per-view, in a costly experiment called QUBE.

Another service introduced in 1978 in Los Angeles called SelecTV offered impulse pay-per-view programming. To buy a program, subscribers pressed a green button on the set-top box which authorized the unscrambling of the signal and stored details of the purchased program. Once a month, each SelecTV set-top box automatically communicated with a company central computer using a telephone device built into the set-top box, and sent in a list of viewed programs for billing.

“Half of my friends told me I was crazy because no one would buy pay-television services because they got television free, and the other half thought I was going to put commercial television out of business,” recalls Robert Block, 88, the company’s founder.

Two-way cable TV communications via set-top boxes eventually became more common, principally to save money by not sending all programming down the pipe all the time but also to expand offerings such as pay-per-view and to thwart piracy. These new back-channels for data also allowed cable operators to sell what is called set-top box viewership data — a business that has grown significantly over the past decade.

With such data, cable providers remove the subscriber’s name but may leave in other details such as a computer IP address before sending it to a data broker, whose big players include Experian and Acxiom. These companies, in turn, figure out to whom the data belongs and append details such as demographic information and shopping histories. Companies engage in this elaborate data dance to comply with a 1984 law on cable television. But the end result is akin to a nurse asking a patient to strip down and put on a hospital gown, only to have the doctor see everything anyway.

Bill Livek, president and executive vice chairman of comScore, a leading audience measurement company which does such matching, emphasizes that individual consumer information data is long gone by the time companies decide ad placement, and at no time is personally identifiable information available commercially. The process is more like making soup, he says, where different ingredients are blended together. “You put salt in, and the salt disappears.”

Still, the result of all this data science are reports segmenting different households into large groups of like-minded consumers so that anyone from a political party to a luxury car maker can easily find potentially receptive audiences. “You can merge auto registration info with set-top box viewing information and then you can start looking at what television shows have a high propensity of folks that may currently own a competitive make and model,” Livek says. “So you can target in on those programs.”

Data sales are not universal among the 660 U.S. cable companies. Some operators do not want to license their information to others, do not have the technology, or have privacy concerns about harvesting it.

The leading audience measurement company, Nielsen, has long relied not only on bulk data from millions of viewers but on a representative sample of the U.S. population. Much as pollsters forecast elections by surveying far fewer voters than the total electorate, Nielsen now tracks about 100,000 TV sets and 50,000 TV-connected devices in roughly 40,000 households, which receive incentives for participating.

Late last year, a new rival, Symphony Advanced Media, started paying a panel of 15,000 people $5 to $13 to download an app that tracks not only TV viewing, but media consumption across social, mobile and PC platforms, says Charles Buchwalter, the company president. They found one unexpected trend: Many people consume multiple sources of the media simultaneously, such as perusing Facebook, sending tweets and watching TV at the same time.

Representative audience sampling has a long history, yet in the era of big data, TV manufacturers, cable, satellite and streaming content providers are increasingly seeking new insights by studying millions of viewers rather than relatively small panels. “What we know from our subscribers is much richer than what one might get from panels or Nielsen,” says Adam Lowy of DISH Network. “There is nothing more powerful than knowing what is going on in that machine.” (Nielsen is at least partially convinced: In April they announced a long-term deal to buy data from DISH).

DISH is also a pioneer in targeting some broadcast ads to individuals based on what they know from outside sources. For example, a Walgreen’s loyalty card member who subscribes to DISH might see a relevant ad from the pharmacy chain, Lowy says. In another recent campaign, the satellite operator aired a mortgage company ad for renters but not homeowners.

Smart TVs, whose popularity has grown dramatically in the past decade, are likely to fuel another dramatic expansion in data collection. That’s because they can log not only video from cable or satellite, but Netflix, Amazon, Roku and other sources that have been reluctant to share their viewership data with others.

“Nielsen might be enough for some companies, others might be looking for different sources,” says Nick Colsey, the vice president of business development Sony Electronics who helped introduce the first Sony Smart TVs at the Consumer Electronics Show in 2007. “We think the TV is a data source that hasn’t been exploited seriously up to now, and now it can be.”

Vizio has led the way in Smart TV data gathering, and wrote enthusiastically about the business potential in an October 2015 filing for an initial public offering. “Our Inscape data services capture, in real time, up to 100 billion anonymized viewing data points each day from our over 8 million VCUs,” it wrote. “Inscape collects, aggregates and stores data regarding most content displayed on VCU television screens, including content from cable and satellite providers, streaming devices and gaming consoles.”

Smart TV manufacturers can learn what their customers watch by adding special hardware to the set and partnering with companies specializing in automatic content recognition, a process that occurs via audio fingerprinting. This method identifies unique patterns in audio and/or video signals seen on a TV screen – whether from traditional broadcasters or a streaming source such as Hulu — and compares them with a central database of fingerprints.

A second possible technique, watermarking, places a video or audio signal into about every second of a program — perceptible only to their software. Because watermarking requires the cooperation of a broadcaster, it is more cumbersome and incomplete. They are not yet widely used by Smart TVs, but are employed by Nielsen in its viewing meters, according to industry officials.

Such techniques require extensive computer processing. Chicago-based 4C Insights monitors over 2,100 channels in 76 countries, and rents a small space in 210 U.S. cities to relay local television feeds to their data center in New Jersey, says Mark Green, the chief operating officer. Some content needs manual human review, which occurs in the Netherlands and in India. The ultimate goal of such efforts is to target and synchronize advertising across television, smart phone, tablets and other devices.

TV manufacturers still face challenges in drawing conclusions from the data they collect. To start with, many homes own a mix of Smart and older TVs that do not transmit data. In addition, each manufacturer only knows what owners of its televisions see and does not know if mom, dad or the kids are watching.

“The part about Smart TVs and collecting data — they are evidently capable of collecting data, but they have the same issue that the cable operators have in that they don’t know the who,” says

Scott Brown, Nielsen’s senior vice president of engineering and strategic relations.

Advertisers could learn a lot more if different manufacturers pooled their data. “It’s a beautiful technology; it’s more elegant than set-top data,” says Bill Harvey, an industry innovator who has worked in media and advertising research since the early 1970s. “But the logical stuff isn’t happening — the assembly of all of the data into one cooperative, which would be such a natural.”

Not surprisingly, some aspects of this brave new world of viewer measurement have stirred concern – when and if the public learns about them. Vizio faces a spate of lawsuits after customers realized that their sets automatically share detailed viewing histories — programs watched, what time of day, channels etc. — unless they opt out of “Smart Interactivity.” Some Smart TV makers do ask buyers to opt into sharing data, which they might call something confusing such as “SyncPlus” or “Live Plus.”

Samsung also faced an outcry last February amid concerns that its voice recognition feature could record private conversations. The policy said: “Please be aware that if your spoken words include personal or other sensitive information, that information will be among the data captured and transmitted to a third party through your use of Voice Recognition.” The company later issued a statement declaring: “Samsung Smart TVs Do Not Monitor Living Room Conversations.”

The Electronic Privacy Information Center (EPIC) wrote a complaint to the Federal Trade Commission on the issue. The group is also concerned more broadly about growing data gathering related to television and video viewing. “The information that these companies are able to glean … is all very sensitive data,” says Claire Gartland, EPIC’s consumer protection counsel. “It produces a very intimate portrayal of a person.”

Amid such criticisms many TV manufacturers and content providers would rather keep quiet about how they use customer data. Executives from LG, Samsung, Vizio, Comcast, Cox, Cablevision, Hulu, Netflix, Time Warner and PBS, as well as several companies involved in automatic content recognition including Gracenote, declined or did not respond to interview requests for this article. Experian and Acxiom also did not share details.

With customer tracking increasingly common across the Internet as well as television, some experts say that openness and choice is the best and perhaps only viable long-term strategy.

“There are ways of doing this where you don’t lose the public trust, where you offer a full disclosure, you offer consumer choice, you don’t try to trick them in any way, you tell them what you are going to do with the data,” says industry veteran Bill Harvey. “If you do anything else it is certainly immoral, if not illegal.”

Adam Tanner is a fellow at Harvard’s Institute for Quantitative Social Science and author of “What Stays in Vegas: The World of Personal Data — Lifeblood of Big Business — and the End of Privacy as We Know It,” which came out in paperback this month.