MSB AssociatesSan Mateo, California

Web Reputation Systems and the Real World: Challenges and Potential Solutions

Introduction

In any effort to anticipate the future direction of society and technology, it is an advisable practice to look to related experiences from the past - successes and especially failures - for important clues. F. Randall Farmer, co-author of Building Web Reputation Systems, shares some of the signposts, warnings, and potential solutions he's encountered over two decades of building reputation systems.

Defining Reputation and Karma

Despite how rapidly digital reputation is expanding in real world influence, it may be surprising to the reader that there is not yet consensus in meaning of this field's fundamental terminology - there is no common grammar or syntax. In Building Web Reputation Systems, Bryce Glass and I attempt to do just that. Rather than echo all of that here, I just need two definitions to point out some of the most important lessons from reputation systems past and present, and to discuss the likely challenges we face going forward.

Today's Web is the product of over a billion hands and minds. Around the clock and around the globe, a world full of people are pumping out contributions small and large: full-length features on Vimeo; video shorts on YouTube; comments on Blogger; discussions on Yahoo! Groups; and tagged-and-titled Del.icio.us bookmarks. User-generated content and robust crowd participation have become the hallmarks of Web 2.0. [Preface] </blockquote> In studying hundreds of online reputation systems, and developing dozens of our own, we found that there was great confusion about the very definition of reputation. When asked about it, most people initially consider only people as the accumulators of reputation - this is odd, given that most public web reputations are associated with products, movies, and restaurants. Upon closer examination, it becomes clear that digital reputation requires a much broader definition than just “the measure of a man.”

We define reputation as any information used to make a value judgement about an object or a person.

But, we still acknowledge that person-centric reputation is still critically important and so we suggest a special term of art for it: karma.

Reputation and Karma-Two Simple Definitions
  • Reputation
    • Information used to make a value judgment about an object or person
  • Karma
    • The reputation(s) for a person

As I mentioned, there is not yet consensus around these, or any other, definitions for digital reputations. This is one of the challenges we face, and it's addressed later in this essay.

Current Practical Applications

Pre-Web Reputation Systems

Modern digital and web reputations are deeply rooted in pre-Internet social systems. In this essay and others in this volume [CITE] examples are drawn from real world models like the Credit Scoring industry and it's flagship reputation score: FICO. But there are many industries that rate their members and suppliers. Cities get ratings for their bonds. Products once received the Underwriters Laboratories (UL) or the Good Housekeeping seals of approval. Consumers Digest has been publishing 5-star ratings since long before anyone had access to computers in the home. Magazines produce unending lists best (and worst) dressed, most eligible, and the richest people in the world. There are many more examples, even before we consider the purely social, word-of-mouth, reputations and advertising influence we leverage to make the hundreds of decisions each day.

But, not all real world reputation systems have continued successfully, or in the same form over time. Looking closely at these systems and how they have evolved may save the developers of digital reputation systems millions of dollars by identifying possible best practices, and more importantly, help avoid dead ends.

Web Reputation Systems

Digital, and specifically web, reputation systems are comparatively new. A good indicator of this is the lack of a consensus on the grammar for talking about them!

Despite any clear cohesive plan, digital reputation systems have exploded. First by making the traditional offline reputation systems, such as credit scores, available on-demand, and next by making nearly every data-driven entity a possible target of reputation algorithms.

The pervasive nature of the web has enabled a global audience to interact with these reputable entities. Once upon a time, it was unlikely that you could gather enough information for a consensus about the quality of some obscure item from your coworkers, friends, and family. Your ability to benefit from reputation was limited by the scope of your social circle.

Now, however, thousands of complete strangers - who presumably share your interests - are publishing their opinions, which are aggregated into digital reputation. Want to buy an antique chess set? Look for a good Seller Feedback score on eBay. What movie to rent from Netflix? 5-star ratings and reviews will do the trick. Want to find out what charities your friends are into? Look at their Likes on Facebook. On the surface, digital reputation systems are just plain better than the old social-local ones.

But, there are some interesting, even troublesome, features and side effects of reducing reputation to digital scores created from the inputs of strangers.

  • Digital reputation scores, by construction, are limited in the subtly of their inputs. Only numbers go in, and those numbers usually come from simple user actions, such as “Marked an Item as Favorite”. One might think that this input would always be interpreted as a positive effect on reputation, but in practice this assumption is incorrect. Marking an item as favorite typically also provides bookmarking semantics, allowing the user to easily return to the item later. It turns out that users who report content as violating a site's Terms of Service often mark the item as a favorite in order to simplify finding out when the item is removed or changed! Thus, digital reputation scores often don't accurately represent the sentiment that was intended.
  • Another class of weaknesses in digital reputation is the ease of manipulation by stakeholders: A cadre of fraudulent New York electronics vendors used to invade Yahoo! Shopping every Christmas, building up false good-reputation for sales by cross-rating each other with 5 stars.

In order for digital reputation to come out of its early experimental and caveat emptor period, we need to acknowledge these issues and address as many as possible before a major economic or political event puts the problems in front of a Senate committee ready to “do something about this.”

The Dreams

Before digging into these challenges, it is important to recognize that the evolution of digital reputation systems continues ad hoc and at an increasing rate. Along with this expanding entropy come the sooth-sayers, whose grandiose visions must also be considered through the filter of experience.

Whenever new technology begins to come into prominence, techno-optimists emerge who see it as a possible solution to one or more contemporary social ills. So it comes as no surprise that many people have invested high hopes in the socially transformative power of digital reputation systems.

Particularly, the ideas that

  • Online karma will displace money in politics
  • Karma should be spendable currency [aka Whuffie]

Though exciting possibilities, these memes require some critical evaluation.

The Rise of Online Karma as Offline Power - Karma Can't Buy Me Love

As more reputation moves online, more karma systems will follow, providing users incentive to continue to make high-quality contributions to web content, to identify users that don't comply with the rules, and to increase the contributor's on personal brand-value.

Craig Newmark, founder of Craig's List, recently posted to his blog [http://www.cnewmark.com] a massively cross-linked article: “Trust and reputation systems: redistributing power and influence” which reflects the thinking of many reputation system optimists: the idea that public karma for users may actually displace other traditional political influence factors. Specifically, a hope that karma will displace money as the measure of power.

This idea - that digital reputation will increase to enable the intelligentsia to rise from the ashes and take their rightful place amongst the powerful - is appealing to technical folks, like us. We've had the Platonic ideal of the Philosopher King with us for more than two millennia. Is it finally time? Will reputation systems enable us to truly identify the people, products, and ideas that will best solve our problems or even govern? Science fiction authors [Ender's Game by Orson Scott Card, EarthWeb by Marc Stiegler, and many others] have suggested this for many years. How far can online reputation take us?

Clearly, digital reputation systems provide a new way to create accountable, numerical scores to millions of people all across the internet. So why not? Shouldn't it be easy for karma to displace monetary political influence?

Good Reputation Has Limited Context

The thinking goes something like ”…we then just combine all the relevant reputations into an overall GoodCitizen or SmartPerson score…” Perhaps combining factors like credit score, large readership/response rate for social media messages and/or blog posts, many excellent endorsements on LinkedIn, etc.?

But, this won't work, as combining scores from numerous contexts makes the calculation about no context at all! It also means that the security weaknesses of the most unsafe contributor-site is the security for the entire score.

Even if you could solve for context and the weakest-link problems, there's the basic problem that any global reputation SmartPerson score then simply represents the result of basic calculations: popularity or industriousness or whatever. Not a whole lot different from the result of a classical election (which is a reputation system.) In the end, winning an election and holding an office sets a single reputation value isElectedOfficial. Not very fine-grained, and in the end highly correlated with the amount of money spent on a campaign. Likewise, online karma systems seem destined to be influenced by money -as the rich pay for the poor to make reputation statements to build a political persons online GoodCitizen karma.

This already happens with movie and business reviews. There are social media marketing companies that are paid to build reputations through posting ratings and reviews. SEO (Search Engine Optimization) is a billion dollar a year industry where individuals, businesses and political parties try to purchase influence.

In short, if money can buy digital karma, the question of money displacing karma in politics is moot. What remains is a different question: Can online karma be a new and additional political force in real life?

But when karma scores are limited to appropriately narrow contexts, they can serve an increasing role within those self-same contexts. Just as real world industry associations tend to be established an operated by those with good reputations within the industry, good sellers on eBay have combined their Seller Feedback karma along with their dollars to influence with the company [and related regulatory bodies] to make needed changes to improve their business. They've even formed the Internet Merchants Association - http://www.imamerchants.org/

Karma as Currency - When Justin Bieber becomes the Philosopher King of the Internet

What about if we turn Karma into something you can spend, like money? Again, a romantic notion: reputation given for good acts gets transformed into currency that you can pay others for them to perform good acts.

Cory Doctorow, in his science-fiction novella Down and Out in the Magic Kingdom, coined the term whuffie to represent karma as a transferable currency. “Whuffie recaptured the true essence of money: in the old days, if you were broke but respected, you wouldn't starve; contrariwise, if you were rich and hated, no sum could buy you security and peace. By measuring the thing that money really represented - your personal capital with your friends and neighbors - you more accurately gauged your success” </blockquote>

A derivative of this idea has been implemented by http://thewhuffiebank.org. It uses traffic at social media sites, such as Twitter, to model influence and creates a score. Basically, if you get retweeted, you gain whuffie, and you transfer whuffie when you retweet others. There is a separate mechanism to explicitly grant whuffie to another user, presumably as a gift or a payment for some product service rendered.

There are several problems with this model: first, it suffers from the same universal-context problem previously defined, only worse: there is no clear way to relatively and reliably set a exchangeable numerical value on based on user actions across contexts. Second, this example turns popularity (something that is already reinforced on social networks with leaderboards and search rank, etc.) into a currency. By extension, WhuffieBank makes Justin Bieber (with almost a million whuffies) the Philosopher King of the Internet!

This problem is not just with Twitter, but systemic to using any karma as currency. It isn't likely to export to real life because popularity and attention isn't a measure of intelligence, trustworthiness, political savvy, technical training, or anything else that might be something useful.

If you decide to create a different currency for each context, say a science whuffie and a sports whuffie and a political whuffie, etc. then you have an accounting and exchange nightmare. How many science whuffies trade for a political one? Does the market float? It quickly becomes an untenable mess.

In short, whuffie - global reputation as currency - crashes on the rocks of complexity due to the universal-context problem: as currencies go, there can be only one (or very few), and by definition there is no truly global context of value. Perhaps the US dollar, with all of its flaws, is the closest thing we'll ever have to a reputation currency.

So? What's it good for?

Though reputation won't become spendable, and universal karma won't displace money (or other forms of influence) in markets or politics, it will become a new and increasing form of influence in its own right. But, so far, that's been an aftereffect: the primary use of reputation and karma remains only within its appropriate and intended context.

The Challenges

Given that digital reputations will increase in real world influence, there are a set of fundamental challenges that should be addressed by those who define and deploy them.

Digital Reputation is not Classical Reputation

Social networks borrow and redefine the metaphor of connecting to a friend to create an edge in a digital graph. This provides a large amount of expressive power for sharing online content. Unfortunately, online the term friend has lost it's real-life subtlety: To give someone access to your social status/content/etc., you must add them as a friend. This mechanism redefines the term. If you're active on Facebook, I'm sure you'll be able to find many people you are connected to that you barely know and wouldn't think of as a close friend in real life. The same kind of oversimplification occurs when we attempt to quantify real-world socially defined reputation.

When we design reputation systems, generally we're talking about numerical scores and small snippets of text. But, compared to real world reputation, these computational models are almost too simple to even merit using the same word. Real life reputation is subtle, personally defined, and internally evaluated. Most of the time, it is inherently non-numerical! On the other hand, a digital reputation's sense of value is concrete (numerical), globally defined for all, brutally simple to calculate, and usually publicly shared - pretty clumsy in comparison.

But it is the utility provided by reputation systems that makes them so worthwhile. Software uses them to sort and filter the unmanageably huge morass of content on the web, so that a human can make the final selection: where to eat, who to buy from, what to watch. It's that last interaction, with a thinking human making a choice, that imbues the reputation score with its real power.

As reputations systems increase in real world influence, the importance of who keeps, calculates, and displays the scores will become a greater societal discussion. Just look at the legal and business frameworks that have evolved around credit scores, such as FICO. But, online reputation systems have been increasingly deployed for more than a decade. Many of the challenges they will face as the scores have ever increasing real-world side effects are already known. By articulating these weaknesses, systems designers and researchers and address them head-on - or enter the fray with their eyes wide open.

Karma is hard

Though digital reputations are often poor copy of the social reputations they attempt mimic, karma is even more challenging In “On Karma: Top-line Lessons on User Reputation Design” a post on BuildingReputation.com, we detailed a dozen ways that karma differs from entity reputation:

  • Karma is user reputation within a context
  • Karma is useful for building trust between users, and between a user and the site
  • Karma can be an incentive for participation and contributions
  • Karma is contextual and has limited utility globally
  • Karma comes in several flavors - Participation, Quality and Robust
  • Karma should be complex and the result of indirect evaluations, and the formulation is often opaque
  • Personal karma is displayed only to the owner, and is good for measuring progress
  • Corporate karma is used by the site operator to find the very best and very worst users
  • Public karma is displayed to other users, which is what makes it the hardest to get right
  • Public karma should be used sparingly
  • Negative public karma should be avoided all together
  • Public karma often encourages competitive behavior in users, which may not be compatible with their motivations

Several of these points will be addressed in this chapter, as the are particularly relevant when considering the possible effects of digital reputation on the real world.

As a harbinger of possible future effects, consider the debate over the use of FICO and other credit scores in the hiring process rages (see Context Limits Reputation). This practice has emerged because the classical job references reputation system has dried up as the direct result of litigation.

When real life reputation systems falter, it seems likely that more and more uses will be found for digital karma.

If Reputation Has Real World Value, Then Methods Matter

The power of reputation systems to provide real, actionable information crowdsourced from large numbers of people and clever algorithms is undeniable and seems will only increase over time. Every top website is using reputation to improve its products and services - even if only internally to mitigate abuse. In short, reputation systems create real world value. Value often translates to wealth, and where there is wealth, the methods for creating that wealth become optimized. Ever since people have trusted the written opinions of others, shady contractors have been generating false letters of recommendations to exploit the unwary.

And the new wealth of digital reputation is created, in large part, by software. This software is usually written under extreme time pressure, and without much design discipline. This leads to two related important challenges that will increase in importance as more and more value is extracted from reputation systems:

  • Reputation system bugs become corporate liabilities. Bad reputation model design or even simple coding errors can lead to innaccurate results that may significantly damage the value of an entity or business. When Comscore discovered problems with its web site ranking algorithm, millions of dollars in web advertising revenue shifted from one company to another.
  • Reputation system abuse becomes lucrative. At some point, the cost to manipulate a score is less than the value created. Spammers learned this more than 15 years ago. Reputation systems are already targets of spammers and hackers! SEO (Search Engine Optimisation) is already a billion-dollar business intended solely to influence search ranking reputation systems at Google, Microsoft, and Yahoo. Likewise, PR and marketing agencies work to create buzz and social networking attention as if they were unbiased individuals.

Potential Solutions

Given the difficulties surrounding digital reputation and the problems of abuse and unreliable code - one wonders who would be brave enough to propose and build a next-generation reputation system! Again, there are lessons from previous systems that can help eliminate, or at least mitigate, the risks:

Context Limits Reputation (especially Karma)

The FICO credit score is most appropriately used to represent the context named creditworthiness and is built exclusively out of inputs that are directly related to the borrowing and repayment of money. The nature of the input matches the nature of the output. We only run into trouble when the score is used for non-creditworthiness tests, such as a pre-employment screening. in fact, according to the National Consumer Law Center, using credit scores for pre-employment screening is an unreliable indicator of productivity and is discriminatory; certain minority groups statistically have lower FICO scores. Likewise, a corporate executive's excellent eBay buyer feedback of 10-green stars has nothing to do with her credit score or with the number of experience points for his Worlds of Warcraft character. Keep the contexts apart.

Don't cross the streams: Good digital reputations should always be context-limited - the nature of the inputs should constrain the use of the reputation scores that are output.

Focus on Quality over Quantity

When applications and sites design reputation systems, they are often looking for an easy method to motivate users to take some action, usually create content. The quickest model to build is in broad use across the web today-participation points give users points for taking actions (usually more points for the more-desirable actions.) Sometimes the points are redeemable (a virtual currency) and other times they simply accumulate indefinitely. These points are often displayed as a karma score, and sometimes are used in competitive forms, such as leaderboards.

If these points are a currency, or they lead to special recognition (such as listing on leaderboards) they often either 1) never catch on because of contextual irrelevance, or 2) lead to the creation of abuse, as described earlier. Purely participation based systems often end up in one of these two states, unless they continually evolve to become more relevant and mitigate abuse. A good example of this is when Digg abandoned it's leaderboards because all of the top contributors were using bots and friends to manipulate their rank.

Taking this problem further, when digital karma increases real world influence or monetary value, it only serves to accelerate abuse.

Instead, it seems likely that the reputation scores that will have the greatest lasting and trusted real world value will be those that are based primarily on inputs based on inputs which I lump under the umbrella term Quality. For sake of simplicity here, a good example of quality reputation input for user-generated content on the web might be Google PageRank score - since that leads to more traffic and could be correlated with advertising revenue.

The best karma scores are generated from indirect input, based on the quality evaluations of the entities that the user has created.

eBay feedback scores are ratings on a single sale transaction - not an overall evaluation of a ongoing history of business with a seller. Yelp is almost the opposite - the business may be evaluated historically. FICO's creditworthiness score is also an indirect karma - creditors report only specific transaction facts, not subjective and non-standardized opinions of the consumer's relationship overall. By tying karma to the quality evaluations of a person's actions, instead of subjective opinions of the person, the score is more reliable and easier to interpret.

Quality evaluations provide increased protection against simple forms of reputation abuse, as it is more difficult to generate large volumes of undetectable indirect feedback. SlashDot uses a scheme to combat indirect input abuse by randomly selecting rated items for a second-level of cross-check, called metamoderation.

Mitigate Abuse through Metamoderation

As any reputation score increases in influence, and especially as that influence has real world side effects, the incentives to abuse the reputation model will exceed the costs of manipulating the score. Note that money doesn't actually have to be involved - social status may be sufficient to generate abuse - game leaderboards, forums, and non-product Q&A sites all have problems with trolls and spammers.

If the community generating the content is sufficiently large - perhaps when moderation costs exceed a certain percentage of the operating budget [say 5%?] - meta-moderation reputation systems have been shown to be effective tools in cleaning up the worst of the worst TOS-violating content. Chapter 10 of Building Web Reputation Systems details a Yahoo! Answers case study where an internal reputation system allowed automatically-determined trustworthy users to instantly remove content from the site. This effectively and completely shut down the worst trolls . Response time to remove TOS violating content fell from 18 hours (using paid moderation) to 30 seconds (with reputation-based meta-moderation) and saved $1,000,000 per year.

If users are creating and evaluating your mission-critical content, consider using reputation-based moderation and metamoderation techniques to identify the best content and also to deal with abuse.

New Solutions, New Problems

So, if future reputation system designers apply narrow context to their data, public-displayed karma is generated based on quality of actions and contributions, and abuse is appropriately mitigated through reputation mechanisms such as metamoderation, are we done?

Even if we take these steps, there are likely to be important new problems to address. These aren't new problems at all, but lessons taken from the social evolution of other real-life institutions that will apply to digital reputation systems as they increase in real-world influence.

The Naming of Names

This chapter's numerous admonitions to limit reputation to appropriate contexts begs the question: what are these contexts? Certainly reputation contexts within a single website can be narrowly and 100% locally defined. eBay Seller Feedback is a good example: its name describes its inputs and its purpose adequately and this deals with the majority of digital reputations. But outside the web, the most influential reputations aren't built solely out of single-supplier input.

What about cross-site, cross-company, and other globally shared contexts, like creditworthiness? How will data be shared across various boundaries? Who, if anyone, will manage the taxonomy of reputation contexts?

Looking at the credit score industry we can see a proven model: the reputation system operator (say Fair Issac Co) creates the context, defines the model, and specifies the data format and API for inputs in exchange for sharing the reputation results. The creditors supply credit history information for their customers in exchange for the right to access the creditworthiness reputation scores of all others, which they use to optimize their business practices.

Is this ad hoc method of identifying reputation contexts sufficient? What are the possible problems?

If cross-domain reputation contexts aren't standardized in some fashion, it seems likely that consumer confusion will result. We see some evidence of this when comparing the 5-star ratings of various product and service sites. For example: Why are the reviews on Netflix so different from Yahoo! Movies? It seems likely that the answer is the context of those who wish to decide what movie to see at the theatre tonight for $20-$50 is significantly different from the context of selecting a DVD, delivered in a few days as part of a $9.99 all-you-can-eat watch-at-home subscription. Other factors may include selection bias and known patterns of abuse on movie sites (where ad agencies post fake positive reviews of first-run films). Even though the formats of the reviews are virtually identical, merging these two databases (and ignoring their contexts) would produce confusing results.

Nonetheless, Facebook's Like system, on the week it was released became the most visible cross-site reputation system on the web. We can expect more companies to follow their lead in producing APIs for remote integration of reputation systems into applications of all kinds, from web sites to social games to mobile phones.

The question remains: should there be an effort to shape or identify the taxonomy of shared reputation contexts? Should there be a set of suggested practices or even requirements for the calculations associated with important real-world impact on reputation scores? By requirements I mean a set of branded guidelines that build consumer trust in these models. In short do we need The Good Housekeeping Seal of Approval for reputation systems?

Privacy and Regulation

As online reputation's influence on the real world increases we will still face the problems of privacy and regulation.

These issues have, to date, been deferred with web reputation systems - perhaps because the inputs were all single sourced, and under the umbrella legal shield of the site's Terms of Service. Generally, allowing a user to completely opt out of a service by deleting their account is seen as sufficient privacy protection and thus far has limited the potential real world legal exposure of the host site. In short TOS defines the context for inputs and reputation, to which the user agrees, and walking away from the data is the escape clause.

But, we're already seeing this perceived corporate protection is eroding on sites like eBay, where some sellers primary income is threatened by the actions of the hosting company on behalf of certain buyers. [LAW SUIT EXAMPLE HERE]. There's the rub-even if the company thinks it is protected by TOS [AND SAFE HARBOR], most reputation systems depend on the actions of other users to create their scores. These three-party transactions are complicated and confusing, and it gets worse when you share data.

Once inputs and scores cross domains, privacy crops up - it's not enough to opt-out of a data source, you need to get your data out/corrected at the data aggregators. Think about a major dispute with a specific credit-card company - perhaps your card was stolen and used to rack up a large sum. After the creditors' laborious flagging process, aggregators have to be notified of disputed items and temporarily remove them from creditworthiness scores - but not all aggregators respond at the same rate or have the same policies. This kind of mess is what leads to calls for state and federal regulation, which can have the disadvantage of codifying in law mis- or under-specified requirements. Exceptions to digital models are truly exceptions and they don't all fit into simply specified algorithms (if they did, they would be rules in the program, not exceptions!)

One way to mitigate government intervention (in the form of regulation) of reputation systems is to establish some sort of industry group to define best practices and conventions for managing the privacy and control of data shared in reputation systems. Industry self-policing is in the best interests of all involved.

Sharing the Data

Though many reputation contexts will be smaller than a single vendor or site (On page xxx, Cascio referrs to this as Constrained network reputation), some providers will want to combine scores, such as email IP-Address blacklists (per Cascio:Universal network reputation.)

The taxonomic, privacy, and regulative challenges facing future digital reputation systems have already been articulated in this and other papers in this volume (see Goldman for much more) - so, how do we minimize the duplicated effort in technology development, policy and taxonomy design, industry standards, reputation modling, and user-interface best practices?

Either we continue reputation system development ad hoc, letting large corporations such as FaceBook establish single-vendor-centric defacto standards and practices for cross context (universal network) reputation, or we use another approach: open standards and open software.

Probably the greatest contributions to the adoption of HTTP and HTML as the backbone data standards for the web were two open source projects: the Mozilla web browser and the Apache web server. They provided the stable frameworks required for others to build a new class of applications upon the internet. Though they weren't bug free, they were the focal point of the majority of effort to produce a reliable system.

In an attempt to provide an open, stable, and common infrastructure for reputation systems development, with the help of [various folks to be named later], I've started the Open Reputation Framework project, part of the [foundation to be named soon] foundation. http://openreputationframework.org

Our hope is to make the Open Reputation Framework home for source implementations of reputation platform implementations, freely released intellectual property, and resources for modeling reputation systems - including a toolkit based on the reputation grammar described in Building Web Reputation Systems.

Assuming that a discussion of best practices related to privacy, regulation, and/or reputation taxonomy need an open and well-lighted place for discussion, might we suggest that the Open Reputation Framework might be an ideal home.

protected/wrsatrw.txt · Last modified: 2010/08/14 12:13 by randy
www.chimeric.de Creative Commons License Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0