# Population of ranked lists

September 29, 2013 10:36 PM Subscribe

I have a statistics question about ranked lists. This is not a homework question.

Let's say I have twenty people who each have their own urn/bucket of balls.

The urn contains labeled balls, painted 1 through 10. The number of balls doesn't matter, except that there a lot of them, and the distribution of labels across all urns is roughly equal, for our purposes.

The labels can also be considered colors, if that helps.

There is dependency between urns, in that a specific ball labeled X in urn A will not sampled from urn B, though there may be other balls in urn B that are labeled X.

The labels correspond to the ranked likelihood of pulling that ball out of an urn, without replacement. We don't know the odds ahead of time, but we know that the odds for choosing a 1 (whatever that label looks like) should be greater than a 2, and 2 greater than a 3, and so on.

Question: We don't know the correct labels ahead of time, and we want to figure out what the correct ranking of 5 labels is from what twenty people sample out of their urns, and estimate proportions.

- Person 1 picks 1000 balls and ranks them in order of the proportion in which they are sampled: {1, 3, 4, 2, 5}

- Person 2 picks the distribution: {2, 1, 4, 3, 6}

- Person 3 picks: {1, 3, 2, 4, 5}

...

- Person 20 picks: {1, 2, 3, 5, 4}

What approach would I use to say with specific statistical certainty that the real rank distribution is {1, 2, 3, 4, 5}, and those labels' odds, based on what all these people sample from their own urns?

Are there tests that deal with this specific problem, starting with population of many ranked lists?

Let's say I have twenty people who each have their own urn/bucket of balls.

The urn contains labeled balls, painted 1 through 10. The number of balls doesn't matter, except that there a lot of them, and the distribution of labels across all urns is roughly equal, for our purposes.

The labels can also be considered colors, if that helps.

There is dependency between urns, in that a specific ball labeled X in urn A will not sampled from urn B, though there may be other balls in urn B that are labeled X.

The labels correspond to the ranked likelihood of pulling that ball out of an urn, without replacement. We don't know the odds ahead of time, but we know that the odds for choosing a 1 (whatever that label looks like) should be greater than a 2, and 2 greater than a 3, and so on.

Question: We don't know the correct labels ahead of time, and we want to figure out what the correct ranking of 5 labels is from what twenty people sample out of their urns, and estimate proportions.

- Person 1 picks 1000 balls and ranks them in order of the proportion in which they are sampled: {1, 3, 4, 2, 5}

- Person 2 picks the distribution: {2, 1, 4, 3, 6}

- Person 3 picks: {1, 3, 2, 4, 5}

...

- Person 20 picks: {1, 2, 3, 5, 4}

What approach would I use to say with specific statistical certainty that the real rank distribution is {1, 2, 3, 4, 5}, and those labels' odds, based on what all these people sample from their own urns?

Are there tests that deal with this specific problem, starting with population of many ranked lists?

Say this part again in different words, please, because it doesn't make sense to me in the context of the rest of your question: "There is dependency between urns, in that a specific ball labeled X in urn A will not sampled from urn B, though there may be other balls in urn B that are labeled X."

The urns are separate, right? There are multiples of each labelled ball within each urn? But all the urns have the same distribution of label probabilities? And the overall, all-the-balls-in-the-universe probability distribution is the same, that is the "true" distribution is 1,2,3,4,5...,10?

posted by gingerest at 11:09 PM on September 29, 2013

The urns are separate, right? There are multiples of each labelled ball within each urn? But all the urns have the same distribution of label probabilities? And the overall, all-the-balls-in-the-universe probability distribution is the same, that is the "true" distribution is 1,2,3,4,5...,10?

posted by gingerest at 11:09 PM on September 29, 2013

Response by poster:

We don't have population counts for each label for each urn, but we know how many total balls we have per urn. When we sample, we get the labels and calculate sample frequencies to do ranking.

There is a "larger, parent urn" from which all the balls are pulled and placed into separate, smaller urns. The balls are assumed to be distributed uniformly to each urn, so that the distributions should be roughly equal per urn within error, but a specific ball put into one urn cannot be put into (or sampled from) a different urn, which could potentially scramble a ranking generated by one person.

posted by Blazecock Pileon at 11:19 PM on September 29, 2013

*Clarifying question: you don't have the actual counts per urn, just sample size and frequency ranking?*We don't have population counts for each label for each urn, but we know how many total balls we have per urn. When we sample, we get the labels and calculate sample frequencies to do ranking.

*But all the urns have the same distribution of label probabilities?*There is a "larger, parent urn" from which all the balls are pulled and placed into separate, smaller urns. The balls are assumed to be distributed uniformly to each urn, so that the distributions should be roughly equal per urn within error, but a specific ball put into one urn cannot be put into (or sampled from) a different urn, which could potentially scramble a ranking generated by one person.

posted by Blazecock Pileon at 11:19 PM on September 29, 2013

The 'urns' here are a red herring - it makes no difference whether 20 people select from a giant urn without replacement, or individual urns, because in either case it's selection without replacement.

posted by Ashlyth at 11:32 PM on September 29, 2013 [1 favorite]

posted by Ashlyth at 11:32 PM on September 29, 2013 [1 favorite]

What this sounds like to me is something from decision theory/ ranking problems.

Have a look at:

http://en.wikipedia.org/wiki/Dominance-based_rough_set_approach#Outranking_relation

http://books.google.com.au/books?id=oY_x7dE15_AC&dq=decision+theory+ranking&lr=&source=gbs_navlinks_s

posted by Ashlyth at 11:39 PM on September 29, 2013

Have a look at:

http://en.wikipedia.org/wiki/Dominance-based_rough_set_approach#Outranking_relation

http://books.google.com.au/books?id=oY_x7dE15_AC&dq=decision+theory+ranking&lr=&source=gbs_navlinks_s

posted by Ashlyth at 11:39 PM on September 29, 2013

So what we really have here is a single variable with ten non-ordered categories of unknown frequency, and we are drawing 1000 individual measurements of that variable twenty times from our population with replacement (or from a population so much bigger than our 1000-individual sample that there's effectively no difference after drawing 1000 samples)? And what we want to know isn't an estimate of the frequencies but the estimate of the relative position of the frequencies?

Now that scientists have some computational muscle, this kind of resampling method is a common way of dealing with estimates where we don't know any of the underlying population distribution parameters. I think what you want to look into is bootstrap methods in categorical data analysis.

posted by gingerest at 11:49 PM on September 29, 2013

Now that scientists have some computational muscle, this kind of resampling method is a common way of dealing with estimates where we don't know any of the underlying population distribution parameters. I think what you want to look into is bootstrap methods in categorical data analysis.

posted by gingerest at 11:49 PM on September 29, 2013

In my experience these things are actually simpler if you just say what you're trying to estimate instead of abstracting back up to people putting balls in urns. In particular, there may well be specific models that have been developed for whatever the actual problem is.

posted by ROU_Xenophobe at 5:48 AM on September 30, 2013 [1 favorite]

posted by ROU_Xenophobe at 5:48 AM on September 30, 2013 [1 favorite]

This problem is usually called "rank aggregation." Formally, it's very similar to voting problems: 20 voters each rate 5 candidates, and you want to infer the "true ranking." There's lots of work on rank aggregation, but off the top of my head, it seems to me that in the case you have in mind the Condorcet method is going to be just fine. For each pair of numbers, like 2 and 3, take a "head-to-head vote" among the 20 people: if more people rank 2 higher, say 2 > 3. (If it's a tie, just say nothing at all, or maybe just use an odd number of rankers...!)

So then you've got a bunch of inequalities. Very probably they fit together compatibly into the ranking you want. (If there are some incompatibilities due to sampling noise, there are various ways to massage those out.)

posted by escabeche at 5:51 AM on September 30, 2013

So then you've got a bunch of inequalities. Very probably they fit together compatibly into the ranking you want. (If there are some incompatibilities due to sampling noise, there are various ways to massage those out.)

posted by escabeche at 5:51 AM on September 30, 2013

"

So long as each bucket is being sampled from identically, each bucket is itself sampled from the same population, and no balls are returned to the buckets providing additional information to take into account, there is no need to worry about them being twenty buckets. One draw from any bucket would be the same as a draw from any other bucket allowing you to think of it as all one big bucket for assessing the larger population.

posted by Blasdelb at 6:10 AM on September 30, 2013

*Let's say I have twenty people who each have their own urn/bucket of balls.*"So long as each bucket is being sampled from identically, each bucket is itself sampled from the same population, and no balls are returned to the buckets providing additional information to take into account, there is no need to worry about them being twenty buckets. One draw from any bucket would be the same as a draw from any other bucket allowing you to think of it as all one big bucket for assessing the larger population.

posted by Blasdelb at 6:10 AM on September 30, 2013

Are you looking for a way to do this by hand or would you be happy with R scripts?

posted by Blasdelb at 6:13 AM on September 30, 2013

posted by Blasdelb at 6:13 AM on September 30, 2013

Yeah but Blasdelb, the question is, does OP have access to the actual ball-counts from each of the 20 people (in which case, yes, they should just be all lumped together) or only the rankings?

posted by escabeche at 6:39 AM on September 30, 2013

posted by escabeche at 6:39 AM on September 30, 2013

I think you could phrase this as a maximum likelihood problem. In the language of the Wikipedia article, θ is a vector consisting of the fractions of each type of ball, and your x

posted by Johnny Assay at 6:50 AM on September 30, 2013

_{i}measurements are the individual rankings. Given a fractional distribution of the balls, there's a given probability f(x_{i}, θ) of getting that ranking. The fact that you just have the rankings, rather than the counts, means that your probabilities f(x, θ) will be difficult (if not impossible) to calculate analytically; probably you'll have to solve the problem numerically instead.posted by Johnny Assay at 6:50 AM on September 30, 2013

*The labels correspond to the ranked likelihood of pulling that ball out of an urn, without replacement. We don't know the odds ahead of time, but we know that the odds for choosing a 1 (whatever that label looks like) should be greater than a 2, and 2 greater than a 3, and so on*

Wait, what? You want the label each ball has to be what the probability is of pulling it out of an urn? You want there to be 10 different labels?

Well, if you have n balls, there's a 1/n chance of pulling a given ball out of the batch. It's the same for all the balls. So if there are 10 different types of balls, and type one has the greatest chance of being picked, you'd need more ones than twos, more twos than threes, etc.

I guess you want each ball to be labeled with a number corresponding to the probability of pulling any ball of the group of balls with that label from the urn. So, make 10 groups of balls, then label each ball with (total number of balls in that group/total number of balls).

*the distribution of labels across all urns is roughly equal, for our purposes*

So there is some mechanism to insure that this happens? If this is about a real-world problem, there needs to be something that insures the balls are put into the urns equally, just assuming it happens isn't going to do that.

*The number of balls doesn't matter, except that there a lot of them...Person 1 picks 1000 balls and ranks them in order of the proportion in which they are sampled: {1, 3, 4, 2, 5}*

So the number of balls does not matter, but there are 1000 balls? And now there are 5 labels, not 10?

*The urn contains labeled balls, painted 1 through 10...The labels correspond to the ranked likelihood of pulling that ball out of an urn, without replacement...Question: We don't know the correct labels ahead of time, and we want to figure out what the correct ranking of 5 labels is from what twenty people sample out of their urns, and estimate proportions.*

If the balls are labeled to indicate what the probability is of pulling them out of an urn, couldn't we just look at the labels on the balls and calculate what their proportions are?

Look, "urns" are often used in probability textbooks to illustrate ideas, but something does not have to be an "urn" for probability to be a useful tool to use in examining it.

Your question is full of internal contradictions. It is IMPOSSIBLE for you to get a correct answer with the information given, because you have stated things about the problem that can't be true at the same time.

I think you need to rewrite your question, carefully, with the order of events as they happen, and what the knowns and unknowns are. It seems like you have tried to fit some other situation to the idea of "urns" and it isn't mapping to that well.

posted by yohko at 11:17 AM on September 30, 2013

This thread is closed to new comments.

posted by PMdixon at 11:07 PM on September 29, 2013