Comments on: How do I process user ratings of content fairly?

Question: How do I process user ratings of content fairly?

Orkboi — Mon, 10 Jul 2006 23:01:13 -0800

How do I process user ratings of content without skewing results of items that have only been rated once?

I'm writing a Web app that allows users to rank entries made by other users from 0 to 4 stars (sort of a "hot or not" for user created content). Users can view a list of items that have gotten the highest average ratings. Now imagine that a new item is created, and someone ranks it 4 stars. It immediately shoots to the top of the list because it's average ranking is 4 stars. Can you suggest a good way to address this problem? I do not want to require items to have a certain number of ratings before they can be listed because I beleive people will want to see their favorites appear in the rankings right away, and I also don't want to discrimate against including something new in the list.

By: Brainy

Brainy — Mon, 10 Jul 2006 23:19:56 -0800

Kinda thinking outloud here but what if you sorted by average rating x number of ratings
(4x1=4) vs. (4x25=100)

If you wanted to be more in depth you could add every rating together so (3x4stars=12) + (1x4stars=4) = 16

Or just sort by rating, then by number of rankings? New stuff would jump to the top layer, but not to the top of the list.

By: sophist

sophist — Mon, 10 Jul 2006 23:37:49 -0800

perhaps something like this:

* = -2 points
** = -1 points
*** = +1 points
**** = +2 points

keep a running total of the points scored per entry, and go off that. items that a lot of people like will have a much higher score than items only 2 or 3 people have voted as liking.

By: grouse

grouse — Mon, 10 Jul 2006 23:38:28 -0800

On kuro5hin, comments are not considered scored until a certain number of people have voted.

By: sophist

sophist — Mon, 10 Jul 2006 23:40:04 -0800

he specifically said:

I do not want to require items to have a certain number of ratings before they can be listed

By: evariste

evariste — Mon, 10 Jul 2006 23:40:15 -0800

Per grouse, most "rating" type websites wait until an item has five or ten ratings before the rating is publicly visible.

By: evariste

evariste — Mon, 10 Jul 2006 23:40:44 -0800

Shit! I'm not the only one with reading comprehension problems, grouse.

By: Galvatron

Galvatron — Mon, 10 Jul 2006 23:45:43 -0800

Let A be the current average of all ratings. For any particular item, compute the displayed rating using a weighted average of A and the item's true rating:

displayed rating = (N*A + K*R)/(N + K)

where K is the number of ratings for the item, R is the item's average rating, and N is a small value that you choose. New items will initially be rated near the average, but the effect of the weighting becomes small as K gets large.

By: delmoi

delmoi — Mon, 10 Jul 2006 23:53:33 -0800

There are lots of ways to do it. The simplest I think is to give everything a 'default' rating with a certain weight. Similar to what Galvatron proposes, but I would use

Rating = D*W_D+∑(R)/|R|

Where D is the 'default' score, W_D is the weight of that score (maybe three? maybe ten?) ∑(R)/|R| Is just a fancy way of saying "the average of the ratings". It's the sum of all the ratings divided by the number of ratings where. A simplified formula would be:

Rating = D*W_D+[average rating]

By: delmoi

delmoi — Mon, 10 Jul 2006 23:57:35 -0800

Hmm, now that I think about it my formula isn't quite right. It should be:

(D*W_D+∑(R))/(|R|+W_D)

So basically what you're doing is giving it W_D 'default' votes.

By: flabdablet

flabdablet — Tue, 11 Jul 2006 03:11:13 -0800

Which is exactly what Galvatron's formula does too.

Simplest way to do this is probably to preload your ratings database with a small number of "neutral" ratings for new items.

By: flabdablet

flabdablet — Tue, 11 Jul 2006 03:14:00 -0800

Sophist's method fails because it ranks items that a lot of people have rated moderately ahead of items that a few people have rated highly.

By: jefeweiss

jefeweiss — Tue, 11 Jul 2006 06:27:55 -0800

I was going to suggest starting an entry with a certain number of average ratings. So when a new entry is created it would have one two star rating automatically and if it got a 4 right away it would only go up to 3 stars.

But the problem with under-rating something right off the bat is that there might be people who only look at high rated items.

By: adamrice

adamrice — Tue, 11 Jul 2006 07:28:45 -0800

Simple solution: show the average rating together with number of votes: that enables people to judge for themselves how much faith to put in the rating.

More complicated solution: assign a confidence rating to every rating based on the number of votes, and normalize the score based on that confidence rating. I'm ashamed to say I'm not good enough at math to work this out, but for any given item, there's a number of votes after which the score will not change significantly (the "confidence interval"). This is your "number of confidence," which we'll call N_C. At one vote or a few votes, every additional score is likely to change the score a lot, for the opposite reasons. As N approaches N_C, you've got more confidence that the current average is an accurate representation of what the average will be when you reach N_C. Perhaps somebody who actually knows statistics can take it from here.

By: baylink

baylink — Tue, 11 Jul 2006 09:49:30 -0800

evariste: it's not that we have reading comprehension problems (which, at least, is a literate insult (pun entirely accidental) :-), it's that there's no really good way to have both behaviors. If you don't weight, and you don't delay, then things rated highly by their first raters will jump to the top; it's the nature of the beast.

You have to compromise *something* if you want to avoid that behavior.

By: grouse

grouse — Tue, 11 Jul 2006 10:29:23 -0800

oops

By: Lanark

Lanark — Tue, 11 Jul 2006 13:21:55 -0800

From memory I think Amazon ratings include an element of time, so new ratings (positive or negative) have less influence than older ratings, the theory being that over time more ratings will appear and balance things out.

By: Mr. Gunn

Mr. Gunn — Tue, 11 Jul 2006 15:37:42 -0800

as adamrice has said, there's no way to do this without adding some other factor to weight the rankings.

To really do it right, any rating given would be weighted by the average rating of the items the rater has himself submitted previously. To bootstrap this in, perhaps people who have rated more items get more weight assigned to their ratings, or perhaps it would be done by number of posts or length of membership.

By: BillyG

BillyG — Tue, 11 Jul 2006 20:00:54 -0800

http://komodomedia.com/blog/index.php/2006/01/09/css-star-rating-part-deux/

I saw this a few months ago but never got it going with a DB to retain the settings, I just checked back today and there is a DB & PHP tutorial added on.

Hope this helps, I will be working on it for my blog tomorrow as I keep seeing it at sites all over the place. G/L.

By: Orkboi

Orkboi — Wed, 12 Jul 2006 20:18:34 -0800

Thanks everyone who answered! This has helped me understand the problem and possible solutions (plus trade offs) much more clearly.