« November 2009 | Main | January 2010 »

December 16, 2009

The Sensical Moment: Asking for User Opinion When the Time is Right

If you're asking for explicit user opinions in your reputation system (ratings, reviews or even just a simple “Like”), pay special attention to exactly when you are asking for them. You'll get better data if you try to gather opinions when it makes most sense to do so: try to find the sensical moments to solicit user input.

Ideally, you'll catch reviewers in moments where they're…

Sufficiently Invested

Can you make it too easy for users to give reviews? You may not think so—if you're in the early stages of deploying your reputation system (or building your site), then you're probably more worried about getting people to use the system at all. And putting obstacles in front of potential reviewers certainly doesn't sound like a good way to alleviate those fears. But, long-term, the success of your reputation system will depend on quality, honest and unbiased opinions.

It may well be in your best interest to limit those who can, and cannot, give ratings. Require that users register, at least. Plain and simple. It should be the bare minimum level of investment that a user should make to voice an opinion on your site.

You may want to go even further. Yahoo! Answers, for instance, limits certain functions (rating questions & answers) to only those users who've achieved a certain status (Level 2) on the site.

Recommendation: Make it easy, but not too easy, for users to give an opinion. Bake in some degree of accountability and ownership for publicly stated opinions.

Appropriately Informed

Don't ask your users to provide opinions on things they haven't experienced. This may be tricky, because the temptation will be strong to make rating objects as easy and low-friction as possible, which typically means putting rating controls in an easy-to-find location and keeping them there consistently. But consider the reputation value of 5-star ratings on YouTube (which we covered here only recently): do you suppose those generally-lackluster ratings distributions would improve if YouTube only allowed users to rate a video after first watching it? (To completion?)

This shortcoming is not limited to YouTube: years ago, Saleem Khan noted a trend on Digg where people were Digging up submissions with no way to have actually read the associated articles. (They couldn't have read them—the articles in question had gone offline before the favorable reviews continued to pour in.)

And even Apple has fallen victim to this oversight. Early iterations of the App Store rating system allowed for anyone to rate an iPhone app—whether they'd ever actually installed the app or not! This violates the "sufficient investment" principle, above, but it also seriously calls into question those reviewers' qualification to review. There's simply no way those ratings could have carried any real value—the reviewers weren't making informed decisions.

Apple eventually fixed this oversight. Now, you're given the opportunity to rate any app from the App Store interface, but when you try to do so for an app you've never tried?


Recommendation: Place ratings inputs either spatially or temporally downstream of the act of consumption.

But Not Overly Biased

Although Apple addressed that problem, they also introduced a new one. Now, when iPhone users attempt to delete an app from their device, they are asked to first rate the app.


This is, of course, a horrible time to ask a user to rate an application. After they've made the decision that they no longer need the app and just as they're in the process of deleting it. Even an app that a user loved may fare poorly under these circumstances.

Perhaps it's truly a horrible app—in which case a bad rating would be justified— or perhaps the user just no longer has any use for it. (Maybe it's a game that he or she has already beaten, or a Twitter client made superfluous by a newer, sexier alternative.) By the time a user is uninstalling an iPhone app, the love affair with that app—if there ever was one—is unmistakably on the wane, and the average ratings likely reflect that fact.

Recommendation: Don't ask for ratings at the low-point of a user's relationship to the rated object.

And not too distracted

Another major sin of the App Store's "parting shot" rating request is that it makes the act of rating into a roadblock. In this excellent comment, PJ Cabrera makes the point:

Who knows how many users are just inputting anything just to move on, without paying attention to what they're doing[?]
True, there is a "No Thanks" button, but its meaning is ambiguous and some reviewers may mistake its intent (perhaps reading it as a "Cancel this deletion" action instead.) It is hard for users to give honest and considered opinions when they are still caught up in the experience that you're asking them to evaluate.

It's common practice, when buying a new car, to receive a customer satisfaction survey from the manufacturer. (This survey is used as an input into the car-selling reputation of the dealership you bought from.) Why do you suppose that the manufacturers will typically wait a week or more before sending you the survey? It's because they know that with a little time and distance from the (often stressful) day of the transaction that you're more likely to give a measured, thoughtful and accurate assessment of the transaction. (You're probably also more inclined to give a positive review, but that's an discussion for another post.)

Recommendation: Respect the primary tasks that a user may be engaged in on your site. Don't interrupt them unnecessarily in order to solicit ratings.

Special thanks to Laurent Stanevich for providing the iPhone app rating screenshot.

December 09, 2009

A Sneak-Peek at Reputation Concepts

Reputation Wednesday is an ongoing series of essays about reputation-related matters. This week, Bryce shares a simple work-in-progress and solicits your input to make it better.

Once upon a time, in (what feels like) a previous life, I illustrated some moderately well-received concept maps: diagrams intended to communicate some simple concepts about software systems and show the interrelationships between their moving parts.

Throughout work on Building Web Reputation Systems, it has always been my intent to attempt a compelling, engaging and fun-to-read concept map. Something to demonstrate the concepts that we've drawn on throughout the book. That was my intent anyway—it just never occurred to me how much work writing a book was going to be. So it hasn't been until fairly recently (like… um, tonight, actually) that I've been able to start pulling something together.

Adhering to our open policy, here, then is that very first rough-and-ugly (and incomplete!) sketch. (Click it for the full version on Flickr.)


I usually don't use Omnigraffle in the design of these concept maps, but it's looseness and speed of idea-capture just felt right for this one, so I'll probably let the general shape of the map simmer for a while in it before moving it over to Illustrator for some fun touches and polish.

This sketch is, admittedly, incomplete. I have a paper version, drafted beforehand, that's easily 150% this size (in terms of # of concepts and linkages.) Please feel free to comment here, or over on Flickr. Hopefully you've enjoyed this brief light interlude, and I'll share more about the progress on the Reputation Systems Concept Map as it evolves.

December 02, 2009

The Cake is a Lie: Reputation, Facebook Apps, and "Consent" User Interfaces

Reputation Wednesday is an ongoing series of essays about reputation-related matters. This week, Randy comes back from the IIW with a simple idea for improving application permissioning.

In early November, I attended the 9th meeting of the Internet Identity Workshop. One of the working sessions I attended was on Social Consent user interface design. After the session, I had an insight that reputation might play a pivotal role in solving one of the key challenges presented. I shared my detailed, yet simple, idea with Kevin Marks and he encouraged me to share my thoughts through a blog post—so here goes…

The Problem: Consent Dialogs

The technical requirements for the dialog are pretty simple: applications have to ask users for permission to access their sensitive personal data in order to produce the desired output—whether that's to create an invitation list, or to draw a pretty graph, or to create a personalized high-score table including your friends, or to simply sign and attach an optional profile photo to a blog comment.

The problem, however, is this—users often don't understand what they are being asked to provide, or the risks posed by granting access. It's not uncommon for a trivial quiz application to request access to virtually the same amount of data as much more "heavyweight"applications (like, say, an app to migrate your data between social networks.) Explaining this to users—in any reasonable level of detail—just before running the application causes them to (perhaps rightfully) get spooked and abandon the permission grant.

Conflicting Interests

The platform providers want to make sure that their users are making as informed a decision as possible, and that unscrupulous applications don't take advantage of their users.

The application developers want to keep the barriers to entry as low as possible. This fact creates a lot of pressure to (over)simplify the consent flow. One designer quipped that it reduces the user decision to a dialog with only two buttons: "Go" and "Go Away" (and no other text.)

The working group made no real progress. Kevin proposed creating categories, but that didn't get anywhere because it just moved the problem onto user education—"What permissions does QuizApp grant again?"

Reputation to the Rescue?

All consent dialogs of this stripe suffer from the same problem: Users are asked to make a trust decision about an application that, by definition, they know nothing about!

This is where identity meets trust, and that's the kind of problem that reputation is perfect for. Applications should have reputations in the platform's database. That reputation can be displayed as part of the information provided when granting consent.

Here's one proposed model (others are possible, this is offered as an exemplar).

The Cake is a Lie: Your Friends as Canaries in the Coal Mine of New Apps

First a formalism: when an application wants to access a user's private Information (I), they have a set of intended Purposes (P) they wish to use it for. Therefore, the consent could be phrased thusly:

"If you let me have your (I), I will give you (P). [Grant] [Deny]"

Example: "If you give me access to your friends list, I will give you cake."

In this system, I propose that the applications be compelled to declare this formulation as part of the consent API call. (P) would be stored along with the app's record in the platform database. So far, this is only slightly different from what we have now, and of course, the application could omit or distort the request.

This is where the reputation comes in. Whenever a user uninstalls an application, the user is asked to provide a reason, including abusive use of data and specifically asks a question to see if the promise of (P) was kept.

"Did this application give you the [cake] it promised?"

All negative feedback is kept—to be re-used later when other new users install the app and encounter the consent dialog. If they have friends who have uninstalled this application already complaining that "If (I) then (P)" string was false, then the moral equivalent of this would appear scrawled in the consent box:

"Randy says the [cake] was unsatisfactory.
Bryce says the [cake] was unsatisfactory.
Pamela says the application spammed her friends list."


Lots of improvements are possible (not limiting it to friends, and letting early-adopters know that they are canaries in the coal mine.) These are left for future discussion.

Sure, this doesn't help early adopters.

But application reputation quickly shuts down apps that do obviously evil stuff.

Most importantly, it provides some insight to users by which they can make more informed consent decisions.

(And if you don't get the cake reference, you obviously haven't been playing Portal.)

December 01, 2009

Pardon our dust...

The book is coming into it's next phase, and we're cleaning up all the messy bits before we hand it off to O'Reilly. This entails renaming every image file and other grubbiness that is bound to break things for a day or two.

Please bear with us while we get things back in order on the blog and wiki.

Bryce and Randy