What's your favorite favorite?
November 18, 2009 6:01 AM   Subscribe

What's the best way to rate comments and why?

I'm doing research into what is the best way to rate comments online. 'Best' being a) it is pleasing and appealing to the user and b) it encourages positive behavior.

Some sites show only positive votes (comment has 100 favorites').
Some sites show both positive and negative votes (100 thumbs up, 200 thumbs down)
Some sites show only the aggregate score (-100 points)

I'm interested in what people find appealing, why they find it appealing, and if there's any commonly-accepted thinking among those who design CMSs regarding which works best.
posted by A Terrible Llama to Computers & Internet (17 answers total) 2 users marked this as a favorite
"Thumbs down" is offputting. If you see that a comment you made has a lot of "thumbs down," or a negative rating, you'll be discouraged from commenting. Now, maybe you made a truly terrible comment and should be discouraged from it, but that's hardly a given. You might have just expressed an unpopular point of view. Minority views enrich debate and should get more, not less, encouragement than crowd-pleasing views. (After all, the crowd-pleasers probably already know their comments will be well-received.) A site with 99% liberal commenters needs the occasional thoughtful conservative comment; it keeps people intellectually honest. And for the truly terrible comments, the better response is (1) comments by others explaining the problem or, if it's bad enough, (2) deletion by moderators. That's why it's better to have the ability to "flag" a comment without visibly subtracting from its rating.

So I'd go with "comment has 100 favorites." That seems pretty unassailable. It's not like it would give rise to an incredibly long discussion with thousands of comments or anything.
posted by Jaltcoh at 6:20 AM on November 18, 2009 [3 favorites]

The most comprehensive I have seen is reddit, where IIRC you can sort by positive votes, negative votes, total votes, or controversial (where both upvotes and downvotes push something toward the top).

I think that the ability to give downvotes leads to an inarticulate disapproving hivemind effect that I find unpleasant. Either something is so pernicious that it deserves deletion (ie. flag it), or one should really be disagreeing verbally or at least picking a disagreeing comment to upvote rather than burying the unpopular opinion with downvotes.
posted by idiopath at 6:24 AM on November 18, 2009

I agree that the feature that hides or greys out 'unfavorite' comments is really not helpful to anyone, as far as dicussions go. Generally they are comments with opinions that go against the perceived consensus, and not really harmful to the subject. It's a cheap way to hide counter arguments. In this case, only positives (like Metafilter, I suppose) is okay. You could also aggregate the best-rated comments somewhere.

Although, I must admit, sites where the discussion is often secondary to the subject (Digg, for example, where most comments are really just people attempting to be witty), plainly ordering by voting is a reasonable approach.
posted by Harry at 6:27 AM on November 18, 2009

Just to clarify, there'd be no action taken as a result of voting (like comments wouldn't disappear if everyone hated them).
posted by A Terrible Llama at 6:31 AM on November 18, 2009

Given the choice, I always prefer a binary system: a comment stands or it gets deleted (by a reasonably impartial and consistent third party). Metafilter's favourites are an acceptable addition to that, but only because the users are relatively sophisticated and the favourites are unobtrusive and have no bearing on the placement of comments.

Everything else is censorship-by-mob. And on a lot of sites, not a very nice mob.
posted by le morte de bea arthur at 6:33 AM on November 18, 2009

On a site like MetaFilter, where there is extensive moderation to get rid of crap, having favorites only is great. But on a site like Digg or Slashdot, downvotes are necessary to hide antisocial behavior that would be deleted by the moderators here. The downvotes lead to ill effects as others have described but I think the only other way is for those to have better moderation, which is labor-intensive (= expensive).
posted by grouse at 6:39 AM on November 18, 2009

I really like Ravelry's system of comment rating, which has "agree" and "disagree" buttons instead of thumbs up/thumbs down, along with "educational," "interesting," and "funny" - so you could rate something as "interesting" and still disagree. I think it discourages favorite/karma grubbing, especially since users' profiles don't show how many total agrees, etc. they've gotten.

You can also click on a user's avatar within a thread to bring up a list of additional options - flag the post, send a direct message, hide their avatar, or ignore all posts by the user.
posted by Metroid Baby at 6:41 AM on November 18, 2009 [1 favorite]

Wilson score confidence interval

posted by A Terrible Llama at 7:14 AM on November 18, 2009 [2 favorites]

I like to see both up votes and down votes, but without it affecting the placement or appearance of the comments. I prefer this because it gives me an idea of where the (local to the interactive portion in that particular discussion) hive mind is trending.

I've seen another site with a rating system like Metroid Baby mentions, and I like that even better, because I can certainly both disagree with a comment and find it compelling or funny. I also think it encourages more ratings, because it is more nuanced, and not just a binary option, and you can rate a comment without flagging or favoriting.
posted by notashroom at 7:22 AM on November 18, 2009

Having mainly recovered from my Wilson-score-induced fainting spell, a question about that model: That is only really necessary in cases where comments being ranked or sorted, is that correct? If I'm wrong can someone draw me a picture?
posted by A Terrible Llama at 7:48 AM on November 18, 2009

I like to see both positive and negative scores.

I wish MetaFilter had an un-favorite flag, too. But I'm a pretty judgmental person at heart.
posted by SLC Mom at 8:53 AM on November 18, 2009

MeFi has up and down votes. Upvotes are shown as "favorites", downvotes are flags that are not shown. You only get one reason for a favorite ("plus!") but several reasons to flag.

When a post or comment gets a number of downvotes (flags), it's usually (?) deleted by a moderator, sometimes with an explanation but usually without. (I'd say "always", but perhaps there are examples of things being flagged to death but left standing, though that doesn't seem to be how it works, based on watching the Deleted Posts.)

This system makes sense enough to my professional head, though there are good arguments for more dimensions ("funny, and saved for later, but I disagree") and for showing the flags, perhaps anonymously if one is afraid of inciting arguments. Because of the hidden-ness of some of the mechanisms here, there are some unwritten "how things work" and "what we like" guidelines going on on MetaFilter that one can only really infer from long, long observation. And even then, you'll never quite have it, because it's a system administered by inconsistent, erratic (but sexy, very sexy!) humans, after all.

As for the value/merits of votes in general, well... there's a sixty-seven thousand message thread somewhere in MetaTalk about favoriting that was triggered when MeFi "temporarily" hid the favorite counts on November 1st, but I don't have the heart to slog through it. It's painful, but discusses all of these things at one point or another, often with great passion. It's also got a few recipes.

MeFi-style systems, with any terminology attached, are a safe common choice: show positives in some way to foster quality, track but don't show negatives as an admin-side QA tool. The mechanism of the (often invisible) negatives varies from place to place, but in general it's all the same goal.

It works.
posted by rokusan at 9:08 AM on November 18, 2009

This is a very touchy area, as that massive MetaTalk thread suggests. I don't think there's a consensus. For what it's worth, however:

I've been on a number of discussion sites over the years, and have experience with everything from completely flat with no ratings (The Well, ILX, MeFi as was), unobtrusive ratings (MeFi up until Nov), obtrusive ratings (Digg, Reddit), destructive ratings (YCombinator) to quite simply batshit insane (Stack Overflow).

In all that time, the best discussions and the best communities have always been towards the no-rating no-anonymity end of the spectrum. Which makes complete sense to me: we have thousands of years of experience in conversation, and in making value judgements without explicitly rating each contribution. It's not only about scale, either, there have been discussions in print that have involved many participants, and they work in fundamentally the same way. Systems that allow us to use this knowledge and experience flourish.

However, the internet also lets us grow communities to the point where they're not really communities at all, they're just aggregations of more-or-less anonymous individuals. They're rabbles, effectively. Ratings have come about as a way of taming these noisy messes, to try and sort the wheat without forcing everyone to read everything. Personally, I think that's fundamentally doomed: you can't engineer society like this; all you do is create an incentive system that's skewed from the usual conversational rewards. Sites like Digg need ratings because they're already broken, and are trying to mould community out of chaos. This is reflected in the quality of discussion there.

As for the "best" way, well, I think the best systems are those that allow us to use our existing conversational skills as fully as possible. That means seeding the community carefully, tending it well, and growing it slowly. There's really not a shortcut to this -- ratings pretend to be a shortcut, but they end up taking you somewhere else, somewhere unpleasant.
posted by bonaldi at 10:18 AM on November 18, 2009 [1 favorite]

Seconding Metroid Baby's comment about Ravelry's system. It's certainly the best one I've ever used; I actually quite dislike most other comment rating systems.
posted by hurdy gurdy girl at 6:47 PM on November 18, 2009

I think what is the best answer for a particular application will depend heavily on why comments are being rates in the first place. What's the problem you are trying to solve by rating comments? You need to know the problem before you can find a solution.

I've found having comments (or rather, book reviews) ordered by how helpful others have found them to be a great feature on amazon, but on other sites I would not care for it at all.
posted by yohko at 6:05 PM on November 19, 2009

Oh, lordy, do I know the problem I'm trying to solve.

I would define the effort as (b) in the post-- "Encouraging positive behavior." Positive behavior being defined as comments which add to the discussion and have value in themselves, that are baseline literate, informative, intelligent, humorous, not racist or sexist or any -ist, not overly aggressive. I'm not looking to 'rank' comments or order them for the user -- it's not really as much for the reader as for the commenter/community.

I'm leaning toward Ravelry's for the community I deal with. I think it's useful in that it most clearly defines "positive behavior" and unlike Metafilter it's a community that really needs that defined.

But I love this stuff in general and could talk about it endlessly.
posted by A Terrible Llama at 3:24 AM on November 20, 2009

« Older Nice shoes, I'm married, she's fine with it, wanna...   |   Quiet Coffee Shop Upper West Side Newer »
This thread is closed to new comments.