« A Sneak-Peek at Reputation Concepts | Main | Waking a Sleeping Chowhound: Another Star-Ratings Misstep? »

The Sensical Moment: Asking for User Opinion When the Time is Right

If you're asking for explicit user opinions in your reputation system (ratings, reviews or even just a simple “Like”), pay special attention to exactly when you are asking for them. You'll get better data if you try to gather opinions when it makes most sense to do so: try to find the sensical moments to solicit user input.

Ideally, you'll catch reviewers in moments where they're…

Sufficiently Invested

Can you make it too easy for users to give reviews? You may not think so—if you're in the early stages of deploying your reputation system (or building your site), then you're probably more worried about getting people to use the system at all. And putting obstacles in front of potential reviewers certainly doesn't sound like a good way to alleviate those fears. But, long-term, the success of your reputation system will depend on quality, honest and unbiased opinions.

It may well be in your best interest to limit those who can, and cannot, give ratings. Require that users register, at least. Plain and simple. It should be the bare minimum level of investment that a user should make to voice an opinion on your site.

You may want to go even further. Yahoo! Answers, for instance, limits certain functions (rating questions & answers) to only those users who've achieved a certain status (Level 2) on the site.

Recommendation: Make it easy, but not too easy, for users to give an opinion. Bake in some degree of accountability and ownership for publicly stated opinions.

Appropriately Informed

Don't ask your users to provide opinions on things they haven't experienced. This may be tricky, because the temptation will be strong to make rating objects as easy and low-friction as possible, which typically means putting rating controls in an easy-to-find location and keeping them there consistently. But consider the reputation value of 5-star ratings on YouTube (which we covered here only recently): do you suppose those generally-lackluster ratings distributions would improve if YouTube only allowed users to rate a video after first watching it? (To completion?)

This shortcoming is not limited to YouTube: years ago, Saleem Khan noted a trend on Digg where people were Digging up submissions with no way to have actually read the associated articles. (They couldn't have read them—the articles in question had gone offline before the favorable reviews continued to pour in.)

And even Apple has fallen victim to this oversight. Early iterations of the App Store rating system allowed for anyone to rate an iPhone app—whether they'd ever actually installed the app or not! This violates the "sufficient investment" principle, above, but it also seriously calls into question those reviewers' qualification to review. There's simply no way those ratings could have carried any real value—the reviewers weren't making informed decisions.

Apple eventually fixed this oversight. Now, you're given the opportunity to rate any app from the App Store interface, but when you try to do so for an app you've never tried?


Recommendation: Place ratings inputs either spatially or temporally downstream of the act of consumption.

But Not Overly Biased

Although Apple addressed that problem, they also introduced a new one. Now, when iPhone users attempt to delete an app from their device, they are asked to first rate the app.


This is, of course, a horrible time to ask a user to rate an application. After they've made the decision that they no longer need the app and just as they're in the process of deleting it. Even an app that a user loved may fare poorly under these circumstances.

Perhaps it's truly a horrible app—in which case a bad rating would be justified— or perhaps the user just no longer has any use for it. (Maybe it's a game that he or she has already beaten, or a Twitter client made superfluous by a newer, sexier alternative.) By the time a user is uninstalling an iPhone app, the love affair with that app—if there ever was one—is unmistakably on the wane, and the average ratings likely reflect that fact.

Recommendation: Don't ask for ratings at the low-point of a user's relationship to the rated object.

And not too distracted

Another major sin of the App Store's "parting shot" rating request is that it makes the act of rating into a roadblock. In this excellent comment, PJ Cabrera makes the point:

Who knows how many users are just inputting anything just to move on, without paying attention to what they're doing[?]
True, there is a "No Thanks" button, but its meaning is ambiguous and some reviewers may mistake its intent (perhaps reading it as a "Cancel this deletion" action instead.) It is hard for users to give honest and considered opinions when they are still caught up in the experience that you're asking them to evaluate.

It's common practice, when buying a new car, to receive a customer satisfaction survey from the manufacturer. (This survey is used as an input into the car-selling reputation of the dealership you bought from.) Why do you suppose that the manufacturers will typically wait a week or more before sending you the survey? It's because they know that with a little time and distance from the (often stressful) day of the transaction that you're more likely to give a measured, thoughtful and accurate assessment of the transaction. (You're probably also more inclined to give a positive review, but that's an discussion for another post.)

Recommendation: Respect the primary tasks that a user may be engaged in on your site. Don't interrupt them unnecessarily in order to solicit ratings.

Special thanks to Laurent Stanevich for providing the iPhone app rating screenshot.


TrackBack URL for this entry: