Comments on: How can I make rankings closer reflect votes?
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes/
Comments on Ask MetaFilter post How can I make rankings closer reflect votes?Sat, 18 Nov 2006 13:05:02 -0800Sat, 18 Nov 2006 13:05:02 -0800en-ushttp://blogs.law.harvard.edu/tech/rss60Question: How can I make rankings closer reflect votes?
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes
In a ranking system, when you have multiple items with similar rankings, how can you give more weight to those with the most votes? <br /><br /> For instance, one item has a 90% rating with only 2 votes, and another has an 89% with 100 votes. Is there a formula that would give more weight, or precedence, to the one with more votes?<br>
<br>
FWIW, I'm using the formula in a PHP script.post:ask.metafilter.com,2006:site.51239Sat, 18 Nov 2006 12:59:16 -0800bjork24rankingformulasPHPBy: kindall
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775343
Sure there's such a formula if you make one up. I'm assuming that you want each total vote for something to be worth say 0.0001 or maybe a little more, in addition to whatever the actual percentage is. So go ahead and do that if that's how you want the ratings to behave.comment:ask.metafilter.com,2006:site.51239-775343Sat, 18 Nov 2006 13:05:02 -0800kindallBy: aberrant
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775344
Showing standard deviation oughta do it.comment:ask.metafilter.com,2006:site.51239-775344Sat, 18 Nov 2006 13:05:29 -0800aberrantBy: Heywood Mogroot
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775346
sdev of 2 votes at .9 and 100 at .89 is the same right?comment:ask.metafilter.com,2006:site.51239-775346Sat, 18 Nov 2006 13:10:20 -0800Heywood MogrootBy: smackfu
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775358
Multiply the rating by (n/n+1). The higher the number of votes, the closer this is to 1.0.comment:ask.metafilter.com,2006:site.51239-775358Sat, 18 Nov 2006 13:20:11 -0800smackfuBy: smackfu
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775361
(Or, if you prefer, you can add that factor. It depends on whether you want to really devalue things with low votes, or just punish them a little.)comment:ask.metafilter.com,2006:site.51239-775361Sat, 18 Nov 2006 13:22:43 -0800smackfuBy: brockerst
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775373
I don't know if it is appropriate in this case, but I think the best way to deal with this situation is to have a minimum number of votes cast, say 10, before the rating is calculated. The information gathered by just 2 votes really has no meaning.comment:ask.metafilter.com,2006:site.51239-775373Sat, 18 Nov 2006 13:39:13 -0800brockerstBy: thirteenkiller
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775384
I played around with this in Excel and the formula I liked best was rank = ((rating out of 100)^2) * ((number of votes)^(1/3)). Excel forumula: [PRODUCT(POWER(A1,2),(POWER(B1,1/3)))]<br>
<br>
That is, the square of the rating from 1 to 100 times the cubic root of the number of votes. <br>
<br>
Here is how this formula ordered the following items with various ratings and numbers of votes:<br>
<br>
1. 95 rating with 75 votes<br>
2. 89 rating with 100 votes<br>
3. 85 rating with 115 votes<br>
4. 85 rating with 50 votes<br>
5. 78 rating with 65 votes<br>
6. 95 rating with 15 votes<br>
7. 45 rating with 500 votes<br>
8. 90 rating with 2 votescomment:ask.metafilter.com,2006:site.51239-775384Sat, 18 Nov 2006 13:53:07 -0800thirteenkillerBy: selton
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775385
Would it be any good for your purpose to round the ratings to the nearest 5% (ie. 81-85% all show as 85%, 86-90% all show as 90%) and then do an additional secondary sort by the actual number of votes?comment:ask.metafilter.com,2006:site.51239-775385Sat, 18 Nov 2006 13:55:51 -0800seltonBy: Lanark
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775394
<a href="http://ask.metafilter.com/mefi/41901">previously</a>comment:ask.metafilter.com,2006:site.51239-775394Sat, 18 Nov 2006 14:04:57 -0800LanarkBy: aubilenon
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775434
How about just adding in a dozen votes for zero for all items, and then use the mean?comment:ask.metafilter.com,2006:site.51239-775434Sat, 18 Nov 2006 15:32:51 -0800aubilenonBy: muddylemon
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775525
Something <a href="http://www.thebroth.com/blog/118/bayesian-rating">like this</a> would work (bayesian ranking formula)comment:ask.metafilter.com,2006:site.51239-775525Sat, 18 Nov 2006 19:22:27 -0800muddylemonBy: rokusan
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775540
For comparison and study, <a href="http://uk.imdb.com/chart/top">IMDB's Top 250</a> formula is:<br>
<br>
weighted rating (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C<br>
where:<br>
R = average for the movie (mean) = (Rating)<br>
v = number of votes for the movie = (votes)<br>
m = minimum votes required to be listed in the Top 250 (now 1300)<br>
C = the mean vote across the whole report (now 6.7)comment:ask.metafilter.com,2006:site.51239-775540Sat, 18 Nov 2006 19:43:19 -0800rokusanBy: Lanark
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#775804
Alternatively you could simply give people both figures, this is what metacritic.com do<br>
<br>
6.3 out of 10 based on 9 votes<br>
7.8 out of 10 based on 73 votes<br>
3.3 out of 10 based on 50 votes<br>
<br>
Its a lot more open and understandable than inventing some random unexplained algorithm.comment:ask.metafilter.com,2006:site.51239-775804Sun, 19 Nov 2006 07:55:27 -0800LanarkBy: DevilsAdvocate
http://ask.metafilter.com/51239/How-can-I-make-rankings-closer-reflect-votes#776703
The IMDB ranking rokusan cites is an example of the bayesian ranking described in muddlylemon's link. To explain what IMDB is doing in words: they are essentially adding in 1300 votes of 6.7 to every movie in addition to the actually cast votes. The "6.7" is simply the average vote over <i>all</i> movies. The 1300 is somewhat arbitrary--pick a number that seems to work well depending on how many votes you have overall.<br>
<br>
<i>How about just adding in a dozen votes for zero for all items, and then use the mean?</i><br>
<br>
This potentially creates the opposite problem: items with few votes being ranked too low. If you also have a "bottom 100" list, should an item with 3 zero votes be ranked lower than one with an average rating of 1.4 based on hundreds of votes? This is way in the bayesian rating, the "extra votes" added in have the value of the average vote over all items, not zero.<br>
<br>
<a href="http://www.boardgamegeek.com/">BoardGameGeek</a> uses <a href="http://www.boardgamegeek.com/wiki/page/BoardGameGeek_FAQ#toc2">both</a> - a true average and a Bayesian average are available for individual games, but the <a href="http://www.boardgamegeek.com/browser.php?itemtype=game&sortby=rank">games sorted by rank</a> list uses the Bayesian average.comment:ask.metafilter.com,2006:site.51239-776703Mon, 20 Nov 2006 09:41:03 -0800DevilsAdvocate