« The Sensical Moment: Asking for User Opinion When the Time is Right | Main | On Karma: Top-line Lessons on User Reputation Design »

Waking a Sleeping Chowhound: Another Star-Ratings Misstep?

Adding new social media features to established communities is always disruptive and not always a good idea.

In Chowhound Comes of Age (For Better or Worse) , Luke Tsai writes about how and why the addition of "industry-standard 5-star ratings" to restaurants on Chowhound.com has quaked the community there.

...Log on to the Chowhound message board for the San Francisco Bay Area and you'll find lengthy threads about where to find, say, the most decadent slice of chocolate cake or the best pajeon (Korean seafood pancakes) in the East Bay. You'll find highly technical analyses of the roasting and brewing methodology of local coffee purveyors.
...Up until fairly recently, one thing you wouldn't find on Chowhound was the kind of star ratings system favored by almost every other restaurant guide, whether in print or on the web — from Frommer's to Zagat to Yelp. On Chowhound, you couldn't give a restaurant any kind of quantitative rating.

In short, it was message boards about food, for and by chowhounds - self selected folks who liked to go off the beaten track to find something interesting to eat. Specifically, they liked to go to the places that were unrated, or rated poorly on other sites, just to find any diamonds-in-the-rough, especially unusual items.

It had no ads, no ratings, no shills (because of strong moderation), and no membership fees. It bootstrapped as a contribution financed community site. Eventually it was sold to CNET, which was sold to CBS, which has added ads and ratings in an attempt to capture revenue.

...Jacquilynne Schlesier, the site's community manager, has been helping to moderate Chowhound since the pre-CNET all-volunteer days. "Our users are incredibly passionate and incredibly knowledgeable," Schlesier says. "But it can be a little daunting if you're someone who's not a long-term chowhound." To help make the process less intimidating, they've revamped the site's restaurant listings — individual pages that have all the basic information about a particular restaurant along with links to relevant discussions on the message boards. It's on these pages that the star-rating feature appears.

Generating revenue is good goal. Most food sites that make money have ratings. Your typical product manager would get this far in their reasoning and implement an industry-standard 5-star rating system. This is what CBS/Chowhound apparently did.

But according to many of the site's devotees, the latest set of changes is particularly "unchowish," in large part because of the star-rating feature. ... Among other criticisms, [the founder of Chowhound] questions how it's possible to "rate a bakery that is horrendous except for one item so great it's worth a 100-mile trip along the same rating scale as a pretty-good diner, an inconsistent high-end sushi place, and an exemplary Italian-ice cart."

This is an excellent point. There is a context mismatch between the discussions (interesting food items) and the rating for a restaurant overall.

Why bother asking Chowhound users for a star rating? It's not like they were clamoring for this feature. This looks like Yelp envy to me. I saw similar lazy product design while at Yahoo! around the time Digg originally exploded in growth - property after property wanted to add "Thumbs up" buttons to everything from the weather to search results. [This was a bad design choice for almost all of them - fortunately, during this me-too frenzy, the legal mess from the posting of the DVD crack key helped most Yahoo! product managers figure out that the Yahoo! audience and Digg's were almost mutually exclusive.]

After spending some time at the Chowhound, I've noticed that those participating in the discussions aren't rating much. I couldn't find a restaurant in my area with more than 5 reviews, and five is probably the absolute minimum number of ratings that should be required for the average rating to mean anything. And even then, the average overall is going to be 4.5 stars - familiar to outside users, sure, but in the end pretty useless as a gauge of quality. And, unless CBS is going to buy ratings from someone else, they will never have enough to be useful in a regional search. Bootstrapping 5-star ratings from scratch is a big mistake.

If not 5-star overall ratings, what else?

Clearly the staff needs to find revenue, and advertising is what they've bet the farm on - so increasing the number of users and user-engagement is required. They had do to something.

But, given just the things discussed in this post, there are several other reputation-based things they could try instead...

1) Let the active board posters determine the context! If it's Best Pastrami Sandwich or Most Exotic Menu - let them give the awards to the restaurant. The simplest implementation of this is tagging, but allowing users to create award categories makes search-ranking easier.

2) Allow discussions/posts to be tagged as well - both with the name of the places that are discussed as well as the same user-generated topics...

3) Allow users to mark a place as a "favorite" which both increases the popularity of the place and puts that place on their profile. Combined with tagging, this is an advertisers dream!

4) Implement a karma system for contributors to discussions, increasing the search-rank value of the businesses they discuss, tag, favorite, etc.

All of these techniques are discussed in detail in our upcoming O'Reilly/Yahoo! Press book: Building Web Reputation Systems, also available in searchable draft form on our wiki.

The Chowhounds have valuable expertise they are sharing, they deserve better tools than a poor copy of every-other restaurant site!


TrackBack URL for this entry: