In the first draft of chapter 6 of our book on reputation system design we begin guiding the reader through a process to create their own reputation model. We start by helping identify the business and other goals they have for their application/site, and quickly move on to what kind control model they have in mind for users creating and interacting with content: What we're calling Content Control Patterns. These patterns have general applicability when talking about all social media design, so we wanted to share them widely and gather feedback as early as possible. We'd especially like suggestions for better names for the patterns.

The individual CCPs (Content Control Patterns) are detailed over on the wiki, so be sure to click through on one of the pattern names or images.

--- Begin Excerpt ---

Whether you need reputation at all, and the particular models that will serve you best, are largely a function of how content is generated and managed on your site. Consider the workflow and life-cycle of content that you have planned for your community, and the various actors that will influence that workflow.

First, who will be touching your communities content-will users be doing the bulk of content creation and management? Or staff? (We'll generically refer to these people as 'staff', but they could be people under your employ, or trusted third-party content providers, or even 'deputized' members of the community, depending on the level of trust & validation that you put into them.)

In most communities, you'll find that content control is a function of some combination of users and staff. Therefore, it's worth our while to dissect the types of activities that each might be doing. Consider all the potential activities around the content lifecycle at a very granular level:

• Who will draft the content?
• Will anyone edit it, or otherwise determine its readiness for publishing?
• Who is responsible for actually publishing it to your site?
• Can anyone edit content that's live?
• Can live content be evaluated in some way? Who will do that?
• What effect does evaluation have on content?
  • Promote or demote its prominence?
  • Remove it altogether from the site?

While you'll ultimately have to answer all of these questions, at this stage these fine-grained questions can be abstracted somewhat. Right now, there are really three questions you need to pay attention to:

1. Who will create the content on your site? Users or staff?
2. Who will evaluate the content?
3. Who has responsibility for removing content that is inappropriate?

There are eight different content control patterns for each unique combination of answering the questions above. Each pattern has unique characteristics when considering what reputation systems you may, or may not, wish to consider for your application. For convenience, we've given each pattern a name: Web 1.0, Submit-Publish, Bug Report, Reviews, Surveys, Agents, Basic Social Media, The Full Monty, but these names are just place holders for discussion, they are not a suggestion to recategorize your product marketing. We will now cover each of these content control patterns in detail so you will understand the implications and ramifications of each.

Web 1.0	Submit-Publish
Bug Report	Basic Social Media
Reviews	Agents
Surveys	The Full Monty

The Content Control Patterns for communities of content. These largely determine the amount, and types, of reputation models that you will need.

If you have multiple content control patterns, consider them all and focus on any shared reputation opportunities. For example, you may have a community site with a hierarchy of categories that are created, evaluated and removed by staff, but perhaps the content within that hierarchy is created by users. In that case, two patterns apply: the staff-tended category tree is an example of the Web 1.0 content control pattern and as such it can effectively be ignored when selecting your reputation models. Focus instead on the options suggested by the Submit-Publish pattern formed by the users populating the tree.

Posted by randy at 05:43 PM | Permalink | Comments ()

We're very excited to announce a new draft chapter is up on the wiki! We've been a little quiet these past weeks as we wrestled with the material for Chapter 2: A Grammar for Reputation because it's so central to the rest of the book. This chapter outlines a visual language for documenting & designing reputation systems (and models, and statements, and…)

Once you, the reader (and um, we the authors—we're still at least partially figuring this out ourselves!) become conversant in the concepts and the language prescribed, we'll use this language throughout the rest of the book to build ever-more involved reputation models and illustrate some well-known case studies such as Digg-style vote-to-promote systems, or some effective abuse mitigation systems that we've worked on during our time at Yahoo!

So, please do review Chapter 2: A Grammar for Reputation and share your thoughts, either as comments on the wiki or here on the blog. This chapter is something unlike what we've published here so far, and we hope that it's an important milestone in thinking about—and talking about—reputation in a systemic and structured way.

Posted by Bryce Glass at 03:33 PM | Permalink | Comments () | TrackBacks (0)

I (Randy) will be leading a session on Reputation and Context at the Online Community Business Forum in Sonoma, Califonia on March 19th and 20th.

My session is currently scheduled to be an hour-long breakout at 2:30 on the 19th. I'm currently thinking of limiting the amount of context-setting material to about 10 minutes and having the rest be a working session for helping people sort out their reputation contexts, models, and abuse mitigation issues. If you're planning on attending, feedback here about what you'd like to see from my session is strongly desired - leave a comment below or drop me an email.

I've attended the ForumOne Community events for several years now and found them to be invaluable. This year should be no exception. The program is still evolving, but already has an impressive list of innovators and stalwarts of Online Communities speaking and leading sessions.

Some seats are still available. Just visit the OCBF Registration page, enter the password sonoma and the discount code farmer at checkout and you will save $150.00

Posted by randy at 02:50 PM | Permalink | Comments ()

Dilbert ©2009, United Feature Syndicate, Inc.

The reputation of the Ratings and Reviews pattern of reputation systems has been taking more hits lately:

• The Daily Background reported: Belkin’s Development Rep is Hiring People to Write Fake Positive Amazon Reviews using Amazon's Mechanical Turk. The going rate was $0.65 each.
• David Pogue soon after wrote Carbonite Stacks the Deck on Amazon as employees wrote positive reviews without identifying themselves as such. This is the account that probably inspired the Dilbert cartoon for Feb. 1, 2009.

Corporate reputation system (specifically Ratings and Reviews) abuse isn't new. See Is Harriet Klausner for real? which deals with an impossibly profuse review writer and Merchants angry over getting yanked by Yelp which details one mitigation technique to shut down review swapping by business owners.

It's safe to say, as long as there is money to be made, all reputation systems - not just Ratings and Reviews - will be subject to this sort of manipulation attempt. The incentive for the corporate user or spammer is clear. Think about it, writing shill reviews is probably significantly cheaper and more effective than sending spam email, at least for now. Yahoo! suffers from an annual attack of this form from November to December as recounted in PriceRitePhoto: Abusive Bait and Switch Camera Store where Thomas Hawk explained:

One of the things that troubles me the most about this situation is that I found this retailer through Yahoo! shopping and they were perceived to have positive feedback. Is the feedback mechanism for Yahoo! Shopping broken? How could this horrible retailer have a four star rating with 858 ratings? I’m convinced that there is a possibility that many of the “reviews” for this company could be fake. I should though have sorted through the reviews to the worst to see that many others had fallen prey to similar fraud by this company.

Oh Thomas, you were right that the good reviews were nearly all fake. There are a dozen or more slimy NYC electronics merchants that have control of hundreds of accounts and they all rate and review each others sites fraudulently. The bad guys have it down to a science and sites like Yahoo! have to detect and remove these abuses every year - something they aren't always very good at.

Publicists and creators write reviews of books and movies that they're promoting. eBay sellers refund purchase prices and even pay cash to get users to remove negative feedback.

Wow, that's a lot of abuse

You almost wonder why people trust these systems at all, but they do. [I'm looking for newer data as I suspect recent increase in the reporting of abuse might erode this confidence, if you have newer info, please comment below.] Clearly, though, it should be a priority to prevent, or at least detect and repair any abuse that a site may have - especially if your brand is associated with higher quality reputation it can translated to higher revenue.

Mitigation Techniques

Strenghten Identity

Require registration to rate and review. Period. Even then cheap identity systems such Yahoo and HotMail, where you can get an email address in a few seconds are at the root of great problems, but once you have an account, you can build up identity strength by having all of the user's significant content interactions attached. Their social network, contacts list, high scores, shared media, profile customization, content contributions, saved preferences and more are time consuming to set up, establishing a switching-cost. The threat of losing this work is deterrent to abuse. The strength of identity can be used to Weight Average Ratings (see below)

Establish Karma

Attaching User Reputation, or Karma to a user provides explicit numeric values that can be used with the other techniques outlined here. See Chapter 8 for a detailed discussion of this topic.

Report only established averages

Reporting something an average 5-stars when there is only one rating is ridiculous. Apply a minimum count before showing the average. You could also do as Amazon does, and surface the distribution of scores. This raises the barrier to entry for review abuse.

Weight Average Ratings

Simply put, more trusted users get more say in the average. New users get (almost) no say, reviews written all on the same day or from the same IP addresses get devalued, etc. Facebook Connect IDs get treated as if they are real people, etc.

Apply a Heavy Hammer

If you detect an abusive account enough delete it, make sure to delete all of its ratings and reviews. Assume they are all tainted. Recalculate all affective averages This is critical to deter an abuser, turning every false review into a Russian-roulette trigger-pull: Will this one kill the account and all the work done?

Community Content Suspension

You can't have eyes everywhere, but your users do. If you implement a system to hide content based on trusted user reports, as Yahoo! Answers did, you can at least get rid of the most obvious stuff nearly instantaneously.

None of these are sure-fire - many of them are just stop-gap techniques until you can build more effective solutions, such as Community Content Suspension or the Heavy Hammer.

Posted by randy at 04:43 PM | Permalink | Comments () | TrackBacks (0)

Up to now, if you'd visited any of the not-yet-drafted Chapter pages over on the wiki, you would've seen... well, nothing! It occurred to Randy and me that this was probably not the best way to solicit public comment. This is particularly problematic because it's exactly at this stage—before we've drafted a chapter—when comments and direction are most valuable.

So today Randy migrated most of the content over from the proposal that we presented to Yahoo! and O'Reilly for the book, so each incomplete chapter now features a SUMMARY direct from our book proposal. These summaries are sketchy outlines for the chapters contents that hopefully call out concepts that we're anticipating.

Please check them out, and add comments with suggestions for additions, or questions on what's already there. This guidance is invaluable as we embark on each new chapter. Here are the incomplete chapters, to get you started:
Chapter 2: A Grammar for Reputation
Chapter 3: Execution Environments for Reputation
Chapter 4: Basic Building Blocks
Chapter 5: Simple, Common Models
Chapter 6: Consider Your Goals
Chapter 7: Objects, Inputs, Scope & Mechanism
Chapter 9: Application Integration, Testing & Tuning
Chapter 10: Keeping Your Reputation Community Healthy
Chapter 11: Case Studies

Posted by Bryce Glass at 09:25 PM | Permalink | Comments () | TrackBacks (0)

We're still refining our approach as we plow ahead with the business of writing Building Web 2.0 Reputation Systems. Late last week, we delivered a draft of Chapter 8 to our editors at O'Reilly and we've been taking a bit of a breather ever since.

Well—a break on the writing: on the process and planning fronts, Randy and I have been expending some energy on dissecting our experience up to now, and trying to tweak some parts of the formula to improve things for subsequent chapters.

Continue reading "Tweaking Our Approach" »

Posted by Bryce Glass at 08:27 PM | Permalink | Comments () | TrackBacks (0)

O'Reilly accepts suggestions of what animal might be appropriate for it's books. We think there are probably a lot of great ideas out there for Building Web 2.0 Reputation Systems, so we've opened up a wiki page to gather your suggestions. Perhaps, if you hit upon the perfect creature, it will be selected by O'Reilly. Then again, maybe not, as they maintain editorial control - as is proper. But, this seems to be just the right kind of book to do this particular kind of hive-mind query on... If your suggestion is used, we promise to credit you appropriately.

Posted by randy at 01:06 AM | Permalink | Comments () | TrackBacks (0)

We've just posted the first draft of Chapter 8: Displaying Reputation on the wiki and submitted it to O'Reilly for our first editorial review. We're expecting a lot of feedback about structure, grammar, etc. from them in the coming weeks.

If you're brave enough, we'd love to hear from our peers about the content. So far we've used everything that people have sent us or left in comments to improve our work - and we intend to go right on doing just that. Keep it up, and many thanks!

Below is the first little bit of the new chapter to whet your appetite:

Three Questions

Okay, so you've designed a reputation model and decided how to collect your inputs. But your work doesn't end there. Far from it. No, now you're faced with a number of decisions about how best to use the reputations that your system is tabulating. Specifically, this chapter and the next will discuss your many options for using reputation to improve the user experience of your site, enrich content quality, and help educate and provide incentive for your users to become better, and more active, participants.

We'll walk you through a simple process for deciding how best to use reputations. We'll start with three simple questions:

1. Who will be able to see the reputation?
   • Is it personal—hidden from other users, but visible to the reputation holder?
   • Is it public—displayed to friends or strangers, or visible to search engines?
   • Or is it limited to corporate use—for improving the site or recognising outliers in discrete ways that may not be visible to the community?
2. How will the reputation be used to modify your site's output?
   • Will the reputation be used to filter the lowest- or highest- quality items in a set?
   • Will items be sorted or ranked using it?
   • And/or will this score be used to make other decisions about how the site flows or your business operates?
3. Is this reputation for a content item or a person? There are some fundamental differences in approaches for each.

Posted by randy at 08:58 AM | Permalink | Comments () | TrackBacks (0)

This requires much more attention than I'm able to give it at the moment, but this blog from Ze Frank on fostering online participation looks phenomenally good, and touches on many of the discussion points that we'd like to cover in Bw20RS. Of course, if anyone's qualified to talk about how to engage with your community in a meaningful, and non-demeaning way, it's Ze. So I'd encourage anyone to go check that stuff out, while we work on stealing .. er.. 'assimilating' Ze's lessons into our own flight-plan. :-) (Via Kottke.)

Posted by Bryce Glass at 01:53 PM | Permalink | Comments () | TrackBacks (0)

The companion wiki for Building Web 2.0 Reputation Systems is open, with the initial draft of the first chapter posted for comment and feedback. Next we'll put up the outline and chapter summaries. Expect a new chapter every two weeks, with regular updates of the previous work based on your participation!

Here's a little teaser from the first chapter - Reputation Systems are Everywhere!

Reputation Systems are Everywhere!

A Conversation

Imagine the following conversation—maybe you've had one like it yourself. Robert is out to dinner with a client, Bill, and proudly shares some personal news.

He says, “My daughter Wendy is attending Harvard in the Fall!”

“Really! I'm curious—how did you pick Harvard?” asks Bill.

“Why, it has the best reputation. Especially for law, and Wendy wants to be a lawyer.”

“Did she consider Yale? My boss is a Yale man—swears by their law school.”

“Heh. Yes, depending on who you ask, their programs are quite competitive. In the end, we really liked Harvard's proximity to our home—we won't be more than an hour away.”

“Won't it be expensive?”

“It's certainly not cheap… but it is prestigious. We'll make trade offs elsewhere if we have to—it's worth it for my little girl!”

It's an unremarkable story in the details (okay, maybe most us haven't been accepted to Harvard!) but this simple exchange demonstrates the power of reputation in our everyday lives. Reputation is pervasive and inescapable. It's a critical tool that enables us to make decisions, both large (like Harvard vs. Yale) and small (What restaurant would impress my client for dinner tonight?) Robert and Bill's conversation yields other insights into the nature of reputation, as well…

People have reputations, but so do things

We often think of reputation in terms of people (perhaps because we're so conscious of our own reputation?) but, of course, many things are also capable of acquiring reputations. In this story, Harvard certainly has a reputation, but so perhaps may a host of lesser-considered items. Consider for a moment the restaurant that Bill and Robert are sharing a conversation in; think of the dishes that they've ordered, or perhaps the wine that accompanies their meal.

It's probably no coincidence that they've made this specific set of choices that brought them to this moment: reputation has almost certainly played a part in all of these decisions. This book will describe formal, codeable systems for assessing and evaluating reputations, both for people and things.

Reputations always take place within a context

Bill praises Harvard for its generally excellent reputation (and rightfully so) but this is not what's led his family to choose the school. No, in particular, it's Harvard's reputation as a law school. Reputation is earned within a context—sometimes its value extends outside of just that context (Harvard is generally well-regarded for academics, for example.) And reputations earned in one context certainly influence reputations in other contexts.

Things can have reputations in multiple contexts, simultaneously. In our example, domains of academic excellence, certainly are critical contexts. But geography can define a context as well, and can sway a final decision. Furthermore, all of an item's reputations need not agree across contexts. In fact, it's highly unlikely that they will. It's entirely possible to have an excellent reputation in one context, an abysmal one in another, and no reputation at all in a third. No one excels at everything, after all! For example, a dining establishment may have a 5-star chef and the best seafood in town, but woefully inadequate parking. This can lead to seemingly oxymoronic statements such as Yogi Berra's famous line: “No one goes there anymore—it's too crowded.”

We use reputation to help us make better decisions

A large part of this book will be dedicated to defining reputation in a formal, systematized fashion. But—for now—put simply (and somewhat incompletely) reputation is information used to make a value judgment about an object or person. It's worth examining this assertion in a little more detail.

Where does this information come from? It depends—some of it may be information that you, the evaluator, already possess (perhaps by direct experience, long-standing familiarity or the like.) But a significant component of reputation has to do with assimilating information that is externally produced, meaning that it does not originate with the person who is evaluating it. We tend to rely more-heavily on reputations in circumstances where we don't have first-hand knowledge of the object being evaluated. Where the experiences of others can be an invaluable aid in our decision. (As you will see, this is a critical point when thinking about reputation on the social web.)

What types of value judgments are we talking about? All kinds. Value judgments can be decisive, continuous, and expressive. Sometimes, the judgment is as simple as declaring that something is noteworthy (“thumbs up” or a favorite.) Other times, you want to know the relative rank or a numeric scale value of something in order to decide how much of your precious resources—attention, time, or money—to dedicate to it. Other judgments are less about calculation and more about free form analysis and opinion, such as movie reviews or personal testimonials. Finally, some judgments only make sense in a small social context, such as “All your friends liked it.”

What about the objects and people that we're evaluating? We'll refer to these as Reputable Entities throughout this book (that is, they are capable of accruing reputation.) Some entities are better candidates for accruing reputation than others and we'll give guidance on the best strategies for identifying these.

And, finally, what type of information do we mean? Well, this information could be almost anything! In a broad sense, if it can be used to judge a thing then it informs—in some part—that thing's reputation. When thinking about reputation in a formal, systematized fashion, it's beneficial to think of this information in small, discrete units of existence. Throughout this book, you'll learn that the Reputation Statement is the building block of any reputation system.

... read the rest of Chapter 1 ...

Posted by randy at 12:16 PM | Permalink | Comments () | TrackBacks (0)