As a child, I spent several years of afternoons happily engrossed in "Family Feud," and ever since I have had a mania for polls and surveys. Not only have I always wanted to know what other people think -- this is a basic human trait -- but I have always had a particularly urgent need to know exactly what percentage of other people are thinking it. Since "Family Feud" asked contestants to guess the results of a survey of 100 Americans, (contestants would be presented with questions like, "Name an animal likely to be found on a farm," and would score the most points for guessing the most common answers), my vision of utopia, is knowing what 100 Americans think about everything, all the time: what percentage of them like the new Fox sitcom; what percentage think Kenmore dryers chew up their clothes; and what percentage think Margaret Thatcher is still the prime minister of Great Britain.
My little fantasy of having instant access to a compendium of opinion surveys might have been preposterous before the Web, and yet in just a few months we have gotten much closer than we ever previously imagined to my peculiar vision of a plebiscite society.
When Amazon.com launched its online bookstore, its most striking feature (besides the simple fact that it let people buy books online) was that Amazon encouraged users to rate and discuss books in what amounted to virtual bookstore aisles. Being able to find out what other readers think about books that you're thinking of buying was a natural and enticing use of the Internet.
Amazon itself has extended the ratings idea to a host of products besides books. And the initial idea has spawned a whole new sector of the Web: sites devoted to consumer ratings.
Consumers instantly reaching a wide audience with their take on whether their money was well spent is shaping up to be one of the most dramatic effects of the Internet. For professional marketers, it is undoubtedly also one of the scariest. Remember the advice your mother/teacher/best friend gave you in, oh, about third grade? "Don't get mad, get even"? A lot of people now have a way to get even with companies they believe gave them a bad deal -- and they see getting even as a public service.
Take Greg Plough, a onetime Prodigy customer who was so dissatisfied with the service he got from the Internet access provider that he posted a dismal review online. "I wanted to send out a warning to the millions of people who are getting shafted by [Prodigy's] rebate deal," he wrote on one consumer review site.
The most interesting twist to consumer ratings, however, comes from sites like Deja.com and Consumer Review. They don't simply rate products (or, in Deja.com's case, not just products, but pretty much everything -- from magazines to political candidates to cat breeds) but assemble the ratings into neat lists of the best and the worst.
A lot of the fun of rating things -- most of the fun, actually -- is that once you've rated them, you can rank them, too. Let's face it, we don't just want to know whether a particular movie is good or whether a particular video recorder is reliable. No, we want to know its particular and very precise degree of goodness, and before we shell out our money we want to know if that camcorder's degree of goodness is even a shade of a fraction less than some other model's.
Both Consumer Review and Deja.com distill user comments into an average rating, from 1 to 5. Deja.com does it in an especially elegant way, copying the "Zagat Survey" approach to rating restaurants by creating four measures in each category (like "comfort," "features," "reliability" and "cost" for sedans, or, a bit strangely, "delivery," "interview style," "reporting" and (!) "sex appeal" for television newscasters) to create a veritable feast of numerical ratings. Deja.com links from the product ratings directly to retailers and Consumer Review has plans for a similar system. Both sites are betting that consumers will use the ratings to make purchase decisions -- and entitle the sites to a cut of any sales made.
If, like me, you loved "Family Feud" and are a sucker for the nifty chart at the back of Entertainment Weekly which compresses the results of a dozen different critics' responses to a movie into one handy little chart, you'll probably also find yourself a junkie for the survey results at Consumer Review and Deja.com. But the more you look at the ratings, the more you are likely to notice some odd results.
The first peculiar result of the proliferation of online surveys is what I have come to think of as "the Philadelphia effect." The term is mine, but credit for noticing it goes to Tim Zagat, the co-creator (with his wife, Nina) of the Zagat guides.
The Philadelphia effect takes its name from a survey conducted some time ago by a glossy magazine. "There was a magazine that surveyed its readers about what was the best restaurant city in America," recalls Zagat. "The result was Philadelphia. I almost choked."
Of course, there are undoubtedly people (most of them, presumably, native Philadelphians) who will be offended by Tim Zagat's blithe dismissal of the opinions of hundreds of magazine readers. Who is Zagat to say that Philadelphia isn't the best eating city in America?
If, however, you do not live in Philadelphia, you might be inclined to the view that the fact that Philadelphia was named as the best dining city in America says a lot less about Philadelphia or its restaurants than it does about the Philadelphians who we can safely presume are the restaurants' most frequent customers. It might well mean not that the restaurants are the best, but that the patrons are among the least demanding. The ratings wind up telling you a lot more about the raters than they do about the city or its restaurants.
Online, it's really easy to see the Philadelphia effect in action. Consider, for instance, the Deja.com ratings for beer.
Even if you are not a beer drinker, it is likely that you know that supermarket shelves are filled with microbrews like Pete's Wicked Ale or Samuel Adams, and plenty of high-priced imports like Guinness. Many of these are quite good. Expensive, but good. And some of the same beers that win competitions also score well in Deja.com's ranking.
But up at No. 2 in Deja.com's rankings is an inexpensive macro-brew much loved by college students: Rolling Rock. (The top spot goes to Yuengling Lager.) I know a lot about Rolling Rock because in college, when I drank beer, Rolling Rock was the beer that I drank. And it's not a bad beer. But the second-best in the country? Better than all the beers that sell at three times the price? Well, maybe. But not very likely. In fact, it's a good bet that even most of the people who praised Rolling Rock on the site, would not rank it so highly in a side-by-side comparison.
Of course, it's hard to prove that Rolling Rock isn't the second best beer in the country, just as it's very hard to prove that Philadelphia doesn't have the country's best restaurants. But it's an awfully suspect result.
The Philadelphia effect has a corollary, which you can think of as the Alan Keyes tsunami. The Philadelphia effect is the result of a lot of people, who are easy to satisfy, giving a surprisingly high rating to Philadelphia restaurants or Rolling Rock beer. The Alan Keyes effect is the result of a lot of people who have a very strong opinion on one product (or, in Alan Keyes' case, one political candidate) sign on en masse and flood a site with reviews.
The Alan Keyes tsunami gets its name from a Republican presidential candidate. If you haven't heard of him, you're not alone. In August's Iowa straw poll, Keyes got only 5 percent of the vote among a collection of Republican die-hards. Poor Alan Keyes is on his second run (very likely you didn't hear about the first one either), gamely shambling between pancake breakfasts with nary a television camera in sight, but at least he has this consolation: Deja.com's raters are firmly in his camp.
In fact, Keyes scores highest of all the presidential candidates rated on the Deja.com site. He is followed by John McCain, Bill Bradley, George W. Bush, Steve Forbes; Jesse Ventura, the former wrestler turned Minnesota governor scored higher than Keyes -- but he's not even running. Keyes' popularity is no statistical fluke. At last count, 1,971 people put in scores for Keyes -- fewer than the nearly 4,000 who expressed an opinion about George W. Bush -- but a very respectable sample pool nonetheless.
Even the people behind Deja.com seem to take some of the ratings with a grain of salt. "We're definitely not the U.S. Census," cautions Deja.com executive vice president David Wilson.
Results like this can make old-school experts in the art of consumer surveys apoplectic. Says Zagat: "A lot of people who say they are doing surveys are just gathering a lot of comments." Zagat points out, for instance, that professional surveyors will often poll a very large group of people and pull out a much smaller, demographically balanced sample to get more useful numbers.
You can ask a lot of people a question, but there's no guarantee you will get a sensible answer. But while this fact of Web surveys might leave some experts and traditionalists aghast, consider this: We love surveys and ratings not so much because of what they tell us about the product, the movie or the candidate at issue, but because they tell us about the people doing the ranking. If the ratings were always accurate, we would surely be disappointed. After all we do not read them just to get good advice, but often read them (just as we watched "Family Feud") for the delicious chance to disagree with the numbers and complain about how silly they are.
Shares