Main | January 2009 »

December 21, 2008

Leaderboards Considered Harmful

Note: Randy and I both had the good fortune to be asked by our (ex- and current) Yahoo! colleagues Christian Crumlish and Erin Malone to provide essays for their upcoming book, Designing Social Interfaces. Randy's essay, The Tripartite Identity Pattern is available on his own blog. This essay appears in draft form here for the very first time. Public comment is welcome and encouraged! Help me improve it in time for inclusion in Erin and Christian's (sure-to-be-awesome) book.

It's still too early to speak in absolutes about the design of social-media sites, but one fact is becoming abundantly clear: ranking the members of your community—and pitting them one-against-the-other in a competitive fashion—is typically a bad idea. Like the fabled djinni of yore, leaderboards on your site promise riches (comparisons! incentives! user engagement!!) but often lead to undesired consequences.

So why do we use them? The typical thought-process goes something like this: there's an activity on your site that you'd like to promote; a number of people engaged in that activity who should be recognized; and a whole buncha other people who need a kick in the pants to jump in. Leaderboards seem like the perfect solution. Active contributors will get their recognition: placement at the top of the ranks. The also-rans will find incentive: to emulate leaders and climb the boards.

And that activity you're trying to promote? Usage should swell with all those earnest, motivated users plugging away, right? It's the classic win-win-win scenario! In practice, employing this pattern has rarely been this straightforward. Here are but a few reasons why leaderboards are hard to get right.

What do you measure?

Many leaderboards make the mistake of basing standings on only what is easy to measure. Unfortunately, what's easy to measure oftentimes tells you nothing at all about what is good. Leaderboards tend to fare well in very competitive contexts, because there's a convenient correlation between measurability and quality. (It's called "performance"—number of wins versus losses within overall attempts.)

But how do you measure quality in a user-generated video community? Or a site for ratings and reviews? It should have very little to do with the quantities of simple activity that a person generates (the number of times an action is repeated, a comment given or a review posted.) But these types of things—discrete, countable and objective—are exactly what leaderboards excel at.

Whatever you do measure will be taken way too seriously

Even if you succeed in leavening your leaderboard with metrics for quality (perhaps you weigh community votes, or count 'send-to-a-friend' actions), be aware that—because the leaderboard singles these factors out for praise and reward—your community will hold these things in high esteem as well. Leaderboards have this amazing 'Code of Hammurabi' effect on community values: what's written becomes the law of the land. And you'll likely notice this effect in the things that people do—and won't do—on your site. So tread carefully—are you really that much smarter than your community, that you alone should dictate the makeup of its character?

If it looks like a leaderboard, and quacks like a leaderboard…

Even sites that don't display overt leaderboards may veer too closely into the 'comparative statistics' realm. Consider Twitter, and its prominent display of community members' stats.

garymoneysmith

The problem may not lie with the existence of the stats but—perhaps—in the prominence of their display. They give Twitter the appearance of a community that values popularity and the sheer size of your social network. Is it any wonder, then, that a whole host of community-created leaderboards have sprung up to automate just such comparisons? Twitterholic, Twitterank, Favrd and a whole host of others are the natural extension of this value-by-numbers approach.

Leaderboards are powerful and capricious

In the earliest days of Orkut (Google's also-ran entry into social networking), the property managers featured a fun little widget at the top of the site: a country-counter, showing members' geographical origins. Cute, right? Harmless, certainly. Google had no way of knowing, however, that the entire population of Brazil would make it a point of national pride to push their country to the top of that list! Brazilian blogger Naitze Teng writes: "Communities dedicated to raising the number of Brazilians on Orkut were following the numbers closely, planning gatherings and flash mobs to coincide with the inevitable. When it was reported that Brazilians had outnumbered Americans registered on Orkut, parties [...] were thrown in celebration."

Today, Brazil maintains its number one position on Orkut (51% of Orkut users are Brazilian as of this writing—the US and India are tied for a distant second with 17% apiece.) Orkut is—basically—a Brazilian social network. Which is not a bad "problem" for Google to have, but probably never an outcome they would have expected from such a simple, small and insignificant thing as a leaderboard widget.

Cui bono?

This may be the most insidious artifact of a leaderboard community: the very presence of a leaderboard changes the community dynamic and calls into question the motivations of everyone for any action they might take. If that sounds a bit extreme, consider Twitter: friend counts and followers have become the coins of that realm, and when you get a notification of a new follower...? Aren't you just a little more likely to believe that it's just someone fishing around for a reciprocal 'follow'? Sad, but true. And this is a site that itself has never officially featured a leaderboard. Twitter merely made the statistics known and provided an API to get at them: in doing so, they may have let the genie out of the bottle.

Meet Bryce

Bryce Glass is a Sr. Interaction Designer at Yahoo! Inc, and has worked on Internet community products and platforms for most of the past 10 years, with some of the Internet's best-known brands (Netscape, America Online and Yahoo!)

Bryce was the User Experience lead for Yahoo's Reputation Platform and consulted with designers and product managers on a number of properties (Yahoo! Buzz, Yahoo! Answers and Message Boards, amongst others) that employ user- and content-reputation. Bryce distilled the research and best practices from those engagements into a series of User Experience Patterns for Reputation that have garnered much attention in the social software design community.

Bryce is also recognized for his work on visualizing complex ideas in a straightforward and approachable fashion. His diagram, Flickr User Model, has been featured in numerous information visualization blogs and was included in Dan Brown's Communicating Design. He looks forward to the challenge of illustrating a deep and detailed model for reputation systems in this book.

December 19, 2008

Meet Randy

F. Randall “Randy” Farmer has been creating online community systems for over 30 years, and has co-invented many of the basic structures for both virtual worlds and social software. His firsts include: one of the first multiplayer online games; one of the first message boards; the first virtual world; the first avatars; the first online marketplace; the first user newsfeed/friend feed (in Yahoo! 360°); the first multi-purpose reputation platform and grammar; and many other smaller firsts—several of which are documented in the form of various patents which are either granted or in process. He has co-authored numerous papers on the topics of virtual worlds and social media which have been published in books and across the Internet. His most widely read publication is entitled The Lessons of Lucasfilm's Habitat, which he co-authored with Chip Morningstar. Lessons is cited in over 50 publications (according to Google publication search.)

For almost 5 years Randy worked as the community strategic analyst for Yahoo!, the world's largest Internet portal, advising Yahoo properties on best practices for construction of their online communities. Randy was the principal designer of Yahoo's global reputation platform and the reputation models that were deployed upon it:

  • Yahoo! Mail antispam
  • Yahoo! Answers troll and quality control
  • Yahoo Message Boards message quality end-user karma
  • And others…

Randy was also a co-author of the Yahoo Open Strategy initiative. After the completion of this book on reputation, he hopes to publish another book on the Yahoo! Open Strategy and one on his experiences from three decades of online community building, tentatively entitled: Context is King.

Randy is in high demand as a freelance social media/community systems design consultant and public speaker, and his online publications and interviews are widely read and cited.

A Reputable Blog

Hello, and welcome to our brand-new blog, which is a companion site to our in-progress book: Building Web 2.0 Reputation Systems. "We," in this case are F. Randall (Randy) Farmer and Bryce Glass. (Of course, we'll introduce ourselves in a little more detail soon.) The book will be a Yahoo! Press release, published by O'Reilly Media.

As the name would suggest, Building will be all about the design considerations (technical-, community- and user-experience-related) of deploying reputation systems for social media sites. It's a topic that we feel passionately about (and one that's timely) and our hope is that this blog—and an accompanying wiki—will afford you, our gentle reader, the opportunity to engage in this process with us.

Our editors at O'Reilly have basically given us carte blanche to share material from the book well in advance of publication, in whatever format we see fit. We anticipate that this may be a messy process (at least at first) and possibly not suitable for the faint-of-heart.
Some of you may ask "How open could you possibly be? Why would I buy the book if all the material is out there on the Web?" Our answer to this is simple: with this site and the accompanying wiki, we're making an earnest attempt to embody the principles of the Unbook philosophy.

By becoming part of this community and participating in any fashion (comments are welcome, edits to wiki pages even better!) you should be prepared to see the sausage being made. In fact, we hope you add a liberal dash of your own spice! At some point (Fall of 2009, God-willing!) all of this effort will yield a tangible, physical, and nicely-formatted benefit: a hearty meal that anyone can sit down and enjoy.

So don't grab that napkin, just yet—no, first let's us reach for some aprons...