Ranking Method

[jazzmatazz2005]

I have a site that allows my members to rate people in three different categories.
all the categories you can rate people from 1 to 5
and if you rate someone you have to rate them in all three categories.

I want to be able to give all the people that have been rated a rank.

I have tried several different ways but none of them seem to work properly

right now im working with getting the overall average, but this doesn't work because multiple people can have a average of 5 and this makes it hard to rank that person.

Any ideas that may work are welcome

[wige]

With a 1-5 ranking system, there is only so much you can filter by - after all, you have a maximum of fifteen points that a user can gain. The only way to add an additional factor might be to incorporate the number of rankings (someone that got 15 points from 1 user might rank lower than someone that got 15 points from 50 users) or the rank of the ranking person (ranks from someone with a rank average of 1 might count for less than from someone with an average of five).

[coachroth]

another option to rank those with an identical average is to check the average points a ranker or rather the rankers are giving:
I.e. Person A and Person B are both getting a straight 5, but those who ranked Person A, gave an average of 12 points each Person they ranked while those ranking Person B gave only an average of 9. Wouldn't you say that the 5s of those ranking Person B are more valuable than those 5s of the rankers of Person A.
Of course this is also just a rough approach which can be fine-tuned a lot.
Just my 2ct

[ronchalice]

Another option to add into the mix that might be helpful is to have them rank their top five (or three, or ten, etc) choices after they have rated all subjects in the three categories. When combined with the ratings, this has a tendency to bubble the overall candidates up and down into a natural ranking order. If a rater has a tendency to rate everyone high or low, then the forced ranking will break people out of those clumps of identical scores.

[arius]

High only, or low only raters are problem. That is why I would recommend using a standard deviation of ratings per rater. That way, you can deemphasize or even remove raters that rate everybody 4 or 5 (or 1 or 2 for that matter).

[Scout]

Assign points to each of the 1-5 ratings. Base rankings on the number of points accumulated in each catagory.



Scout

[Uncle Dog]

Absolute genius!

[xtempore]

Arius is right, you need to deemphasise the opinion of raters who rate everyone the same.

But there will also be people who give nothing but 5's and 1's too, which is much harder to deal with.

One way to cope with this is to consider how close each rater is to the IDEAL rater. An ideal rater should give ratings that fit approximately within a bell curve. Because your values are discrete (1, 2, 3, 4, 5) then you use a Binomial distribution.

So your IDEAL rater should give out their ratings as follows...

1: 6.25%
2: 25%
3: 37.5%
4: 25%
5: 6.25%

(You can tweak these if you like)

Now for each rater you look at their rating bias. For example, let's say we have two raters, called Alice and Bob.

Alice's ratings are like this...

1: 2%
2: 15%
3: 28%
4: 35%
5: 20%

Bob's ratings are like this...

1: 30%
2: 40%
3: 25%
4: 5%
5: 0%

Alice is more generous that the ideal. Bob is quite mean with the points.

One measure of bias would be the sum of squares of the differences, let's call that bias, b.

Of course our IDEAL rater would come out as 0%, and the worst rater (one who ALWAYS gives 1's or 5's) would come out at ~115%.

So...

b(Ideal) = 0%
b(Alice) = 5%
b(Bob) = 14%
b(Worst) = 115%

To adjust these to a weighting we want to give more weight to the lower numbers and none (or little) to the worst raters. Let's call the weighting, w, such that...
w = (115% - b) / 115%

Which then gives us our weightings...

w(Ideal) = (115% - 0%) / 115% = 1
w(Alice) = (115% - 5%) / 115% = 0.957
w(Bob) = (115% - 14%) / 115% = 0.878
w(Worst) = (115% - 115%) / 115% = 0

I'm hoping you know how to then use these weightings to calculate the weighted average for each rating.

There are other ways you could go about calculating the bias and from that the weighting, but the principles remain the same.

[jazzmatazz2005]

i want to thank everyone for their ideas. I have a lot to think about. Ill keep you updated.