Media Metrix reported last week that Lycos pulled ahead of perennial portal champ Yahoo by a small margin in the Web traffic measurement firm's March numbers. That was all it took to boost Lycos'
share price by 36 percent that day.
Media Metrix measures Web sites' traffic by installing software on volunteers' computers (about 40,000 today), tracking their usage and extrapolating the results -- just as the Nielsen TV ratings do. The company blankets the Net with its survey results. Among industry insiders, ad buyers and journalists, it has become the de facto yardstick for who's up and who's down on the Web. And now it has acquired the ability to "move the market" -- the ultimate symbol of power in financial circles.
This is pretty heady stuff for a company that admits its statistics are incomplete -- they're missing significant numbers of users in the workplace and outside of the United States.
"As much as 40 percent of a typical Web site's
traffic emanates from outside the U.S.," according to a white paper
authored by Bob Ivins, senior vice president of Media Metrix. The report goes on
to explain that the company only measures U.S. Web traffic.
At the same time, its panel of 40,000 Web users includes only
7,000 who surf from work -- skewing the results heavily in favor of
home-surfing behavior, despite estimates that workplace surfing accounts
for as much as half of all Web traffic. Server logs from CNN, CNet and
other sites show that the busiest times are during work hours on
weekdays, with peaks at lunchtime. Data from CBS MarketWatch's ad-serving software puts that site's total number of users at "close to double" the Media Metrix figure, says MarketWatch president and CEO Larry Kramer, who figures the sizeable difference comes from the under-representation of working surfers.
The discrepancy between internal server logs and Media Metrix became such
a popular subject within the advertising world last year that FAST, the Future of
Advertising Stakeholders, undertook a data reconciliation study to make
sense of the divergent numbers. Bruce MacEvoy, director of
advertising research at Yahoo and a FAST member, will deliver a report next week comparing panel results to log files from 30 sites -- including
Yahoo, Microsoft and Time Warner -- in conjunction with the ad:tech.SanFrancisco
advertising conference.
"Discrepancies vary based on a given site's attributes, but a big one is
the proportion of users who come to the site from work," says MacEvoy. "No
matter what they say about home and work panels, [ratings firms'
statistics] are best thought of as home-user samples," he says.
At MarketWatch, Kramer says the lack of workplace measurement "is a big
issue." Like everyone in the business, he wants reliable statistics that
not only rank his site high in the Internet lineup but also help
measure trends over the course of the site's history. What he gets,
however, is skewed information. Considering that less than one-quarter of
the overall Media Metrix panel is surfing from work, and the majority of
traffic to the MarketWatch financial information site is during work hours,
Kramer says, there's "a large margin of error."
Media Metrix's numbers are chiefly used by advertisers setting budgets, and that gives these discrepancies real impact. "It's data that is methodologically flawed, but people who spend money use
it to spend money," says Alexander Hughes, group circulation manager for
CNet Online. "It has a huge impact on revenue" for sites across the
industry, he says.
Just about everyone involved in Internet media and advertising is aware that the data are flawed -- but they still use them. Media Metrix numbers are routinely cited in press releases, advertising pitches and presentations to potential investors. It's as if the industry is afflicted by a split-personality disorder: One side knows the numbers are inaccurate, yet the other goes ahead and uses them for the most important business decisions. The justification? We've got nothing better.
"It's important for a site to see its patterns from month to month," says Debbie Newman, vice president of marketing at DejaNews, after explaining that she knows that any statistical sampling can't deliver anything better than an approximation of true traffic patterns. "To me it's not the accuracy of the data, but its value for comparison," she says.
Her comments are echoed by Internet marketers around the country. Even Kramer, who sees such a wide gap between MarketWatch's internally collected numbers and Media Metrix, says the measurement firm's figures "are valuable in that, over time, they track trends." But how valuable is it to know that, for example, you have a higher percentage of women users this month than last month, if neither month's figures are quite accurate?
Media Metrix and its competitors have listened to such criticism for a while, and say they're doing their darndest to correct the problem. But, they say, it's not easy to track Web users who surf on company time.
"Expanding our at-work panel is a huge priority for us," says Media Metrix spokeswoman Michelle Beilsmith. To measure online traffic, Media Metrix and competitor Nielsen//NetRatings require panelists to install tracking software, which reports the user's activities directly to the measurement firm. But it's no surprise that most companies aren't willing to let an outside firm install software on their computers and then monitor the Web usage of their employees.
"We talked to over 170 potential work panelists," says Manish Bhatia, vice president of interactive services for Nielsen Media Research, which joined forces with Net Ratings earlier this year to offer the Nielsen//NetRatings service. "Over 90 percent weren't allowed to install the software, and we ended up with only three people -- they were all small office/home office," he says. In other words, it's no easy feat to come up with a panel that accurately represents the mix of people surfing from small, medium-size and mammoth corporations, across geographical regions, industries and other classifications.
To develop a workplace panel, Nielsen//NetRatings turned for help to CommerceNet, an electronic commerce trade group with about 500 members -- from American Express to UPS to a who's who of Silicon Valley.
In the past six weeks, Bhatia and Tim Meadows, vice president of marketing at NetRatings, have presented their plans for an at-work panel to about 10 CommerceNet member companies. The executives have a lot of questions about corporate security, privacy and software support questions. Nielsen//NetRatings says it will not share results collected about individual workers with the companies, and it will not reveal a company's overall surfing habits. It would only report the information in aggregate form -- say, by industry.
Bhatia and Meadows are looking to sign on a couple of big-name companies, which should attract some smaller ones. Then they'll expand out of CommerceNet and entice other companies to let them randomly choose employees and monitor their surfing habits.
It sounds difficult, though, and it probably will be. After all, how many executives would let some external tracking device inside the company firewall? (Media Metrix does not require that its panelists have the boss's permission.) Still, Nielsen//NetRatings hopes to have a decent-size panel pulled together by the second half of this year.
Sites that get a lot of workplace traffic certainly hope Nielsen//NetRatings or Media Metrix, or someone, will pull it off. "People who are working are in many cases the most attractive to advertisers," says Lynn Gutstadt, vice president of audience research at CNN.
Others, however, argue that no matter how hard online measurement firms work to improve their accuracy, their statistics will never be perfect since they are only projections based on a relatively small number of people's surfing habits -- 40,000 out of an estimated 61 million Web users, in the case of Media Metrix. Many critics say the last thing the Internet needs is this kind of audience measurement -- which was developed for TV.
"Why take a measurement system that's non-empirical and designed to measure broadcast and apply it to the most measurable medium ever?" asks CNet's Hughes. He, along with many others, think that server logs audited by a third party would provide a more accurate traffic picture: "Log files are these perfect Vulcan representations of what people do online."
Critics of log measurement point out that different log analysis practices by different sites make comparisons problematic, if not worthless. Still, even if logs aren't 100 percent accurate, they could come much closer than current projections and track other information, like where visitors come from and when they make purchases.
"The measurement services just count," says Donna Hoffman, professor of management at Vanderbilt University and co-director of Project 2000, a research program on marketing in computer-mediated environments. "But numbers are not that useful when you have the opportunity to know exactly how effective every advertising dollar is."
Hoffman suggests that looking at logs and tracking purchases, click-throughs and other activities could go a whole lot further than audience projections to determine the best use of ad spending: "The Web is a complex multidimensional medium and needs a complex, multidimensional measurement system."
FAST is examining just how to arrive at such a system. Meanwhile, decision makers might consider brushing up on what the numbers they rely on actually mean.
For example, all those Lycos investors may not have realized that Yahoo's reported traffic for March didn't include visitors to the popular GeoCities and Broadcast.com sites it's in the process of acquiring -- while Lycos' figures did incorporate traffic on several sites that it is in the middle of acquiring. In other words, next month's Media Metrix tallies just might show Yahoo besting Lycos again -- and send the markets bouncing, without solid reason, once more.
Shares