Help me with an online ratings system
August 15, 2011 1:56 PM   Subscribe

How to improve an online rating system.

Here are the parameters: It's a system in which people write reviews of plays, giving it one to five stars. The administrators have tried to make the system accountable by asking that people use their real names (or, at least, an ongoing persona), but the system is both easily gamed and trolled. People have friends and family post four and five star reviews, which discourages "meh" reviews and means that, as often as not, the people posting low-star reviews are just being mean-spirited in order to get a reaction. The system suffers in usability as a result -- people wanting an honest audience review of a play instead are plagued by high-star inflation, with almost everything averaging four and five stars.

What are some techniques for addressing this?
posted by Bunny Ultramod to Computers & Internet (4 answers total) 3 users marked this as a favorite
 
The good news is people have done this. The bad news is those people were statisticians.

Ideally, you want more ratings, in hopes that astroturf ratings become noise. There are statistical confidence measures you should be using. It's crazy complex but a good starting point.

Beyond that, I just read some new research on classifying honest vs astroturf reviews. Turns out it's pretty easy to do better than mods, if you can stop focusing purely on quantitative data: low-star reviews, high-star inflation, etc. Those are symptoms. You'll get a long ways towards flagging stuff if you focus on the words.
posted by pwnguin at 2:32 PM on August 15, 2011


Scan for referral source and IP to detect people bringing in the whole family and discount those.

Set up a Bayesian classifier and have manual reviewers sort posts into the kind you want and the kind you don't, and once your system has learned enough you can use it to screen for unwanted reviews.

Allow ratings of reviews.

Write an impassioned plea to respect the culture of the site and police it until members start policing themselves.

Other than that you can't do much, which is why your problems are very common. Good luck!
posted by michaelh at 2:32 PM on August 15, 2011


What they said. Even IMDB has this problem. One guy wrote the IMDB Decoder Ring to help you figure out the actual meaning of a rating. He hasn't updated it in a while, but when last he looked the average rating on IMDB was ~6.5.

I'm pretty sure the code is open source. The blog post explaining the thought process could be helpful.
posted by systematic at 2:59 PM on August 15, 2011


Best answer: Other than that you can't do much, which is why your problems are very common.

Not so. Take a look at this article on Amazon's "billion-dollar button".
posted by mhoye at 6:35 PM on August 15, 2011


« Older Anniversary Ideas (with a baby)   |   Can I go to the US without my US passport? Or... Newer »
This thread is closed to new comments.