# What principle do you use when ranking items? **URL:** \ **Category:** In My Humble Opinion\ **Created:** [December 9, 2022, 6:51pm UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271 "2022-12-09T18:51:05Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![carnivorousplant](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/carnivorousplant/32/3563_2.png) [@carnivorousplant](https://boards.straightdope.com/u/carnivorousplant)\ **Post date:** [December 9, 2022, 11:23pm UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/21 "2022-12-09T23:23:18Z")
> [@thorny\_locust](#): > > Has it occured to them that this would mean that even if all the employees in a particular area happen to be excellent, some of them would have to be ranked as terrible; and even if all of them are terrible, some of them would have to be ranked as excellent? They are business admins. Their thoughts and concepts have little to do with reality. ---
**Author:** ![Eyebrows\_0f\_Doom](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/eyebrows_0f_doom/32/3119_2.png) [@Eyebrows\_0f\_Doom](https://boards.straightdope.com/u/Eyebrows_0f_Doom)\ **Post date:** [December 9, 2022, 11:45pm UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/22 "2022-12-09T23:45:51Z")
> [@Little\_Nemo](#): > > I feel a system which used five ratings conveys more information than a system that uses three ratings. Just as a system that rates things on a scale of one to ten gives more information that a binary system that rates things yes or no. The more possible ratings being used in a system, the more detailed the information it conveys. The system doesn’t use 3 ratings. It uses 5. Your reason for excluding 1 and 5 doesn’t make sense and your “system” doesn’t give any useful information. If you are rating 5 restaurants on whether or not they have good food, your system requires you to give one a 1. Even if they all are excellent and you’d love to eat there again. Therefore your ratings are not of any use to anyone. That _makes no sense_. ---
**Author:** ![Horatius](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/horatius/32/3572_2.png) [@Horatius](https://boards.straightdope.com/u/Horatius)\ **Post date:** [December 10, 2022, 12:18am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/23 "2022-12-10T00:18:27Z")
> [@Little\_Nemo](#): > > Numerical ratings are just an arbitrary system. Yes, but some systems are more arbitrary than other. > [@Little\_Nemo](#): > > I feel a system which used five ratings conveys more information than a system that uses three ratings. But “this item is complete crap/perfect” is important information, and if your scale doesn’t include those endpoints, it’s inherently incomplete. If you insist on using 1s and 5s equally to 2s through 4s, you obscure the endpoints. “Is this really perfect/crap, or just relatively perfect/crap?” ---
**Author:** ![puzzlegal](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/puzzlegal/32/3827_2.png) [@puzzlegal](https://boards.straightdope.com/u/puzzlegal)\ **Post date:** [December 10, 2022, 12:23am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/24 "2022-12-10T00:23:00Z")
It depends what I’m rating. If it’s possible for something to be perfect, then that gets a 5. For instance, an ebay vendor is usually perfect, and back when I used ebay, I mostly rated them “5”. (The thing was what they said it was, and they delivered it promptly.) If it’s something more subjective, like a book or a restaurant, then a 5 means “really excellent”, and I don’t give out many of them. ebay vendor: 5 – perfect transaction 4 – minor problem that was corrected 3 – problems that were ultimately dealt with 2 – serious problems 1 – vendor lies or cheats or steals I don’t think I ever gave any rating other than 4 or 5, and almost all ratings were 5. Movie: 5 – excellent movie 4 – very good movie 3 – this movie has virtues, and some people will enjoy it despite its flaws 2 – this is a bad movie, but there might be reasons to watch it. 1 – I turned it off, and don’t expect anyone to want to watch it. Most movies I see get 4s, with a scattering of 5s and 3s. I don’t see all that many movies, and usually only go to ones that are pretty good. Restaurants: This is trickier, because fancy restaurants, when they are good, are a lot better than the local shawarma place. But the local shawarma place can be a really satisfying meal. I rarely rate restaurants, and if I did, I would want to clarify how I was dealing with that. ---
**Author:** ![don\_t\_ask](https://avatars.discourse-cdn.com/v4/letter/d/e68b1a/32.png) [@don\_t\_ask](https://boards.straightdope.com/u/don_t_ask)\ **Post date:** [December 10, 2022, 12:33am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/25 "2022-12-10T00:33:11Z")
Personally, I feel that 5 point Likert scales only rate about 3 or 4 on a 7 point Likert scale. ---
**Author:** ![Little\_Nemo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/little_nemo/32/3120_2.png) [@Little\_Nemo](https://boards.straightdope.com/u/Little_Nemo)\ **Post date:** [December 10, 2022, 12:46am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/26 "2022-12-10T00:46:16Z")
> [@Eyebrows\_0f\_Doom](#): > > The system doesn’t use 3 ratings. It uses 5. Your reason for excluding 1 and 5 doesn’t make sense and your “system” doesn’t give any useful information. If you are rating 5 restaurants on whether or not they have good food, your system requires you to give one a 1. Even if they all are excellent and you’d love to eat there again. Therefore your ratings are not of any use to anyone. That _makes no sense_. You’re not following what I’m saying. If I’m rating five restaurants and I feel all five of them belong in the top twenty percent of restaurants, then I would give them all a five. And if I felt they all belong in the bottom twenty percent of restaurants, I would give them all a one. ---
**Author:** ![Horatius](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/horatius/32/3572_2.png) [@Horatius](https://boards.straightdope.com/u/Horatius)\ **Post date:** [December 10, 2022, 12:53am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/27 "2022-12-10T00:53:54Z")
> [@Little\_Nemo](#): > > If I’m rating five restaurants and I feel all five of them belong in the top twenty percent of restaurants, then I would give them all a five. And if I felt they all belong in the bottom twenty percent of restaurants, I would give them all a one. Well, at that point we’re just quibbling about what 1s and 5s mean. We seem to be in agreement that, in any real-world collection of items, we could have virtually any distribution. ---
**Author:** ![doreen](https://avatars.discourse-cdn.com/v4/letter/d/858c86/32.png) [@doreen](https://boards.straightdope.com/u/doreen)\ **Post date:** [December 10, 2022, 1:01am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/28 "2022-12-10T01:01:42Z")
> [@Little\_Nemo](#): > > If I restrict the rating of five for things which I judge perfect and the rating of one for things which I judge I’d rather eat broken glass than use (to borrow Horatius’ phrase) then there will be very few things I give a rating of one or five to. It doesn’t have to be “perfect” or “I’d rather eat glass” but giving 20% of the items a 1 , 20% a 2 and so on is sort of a bastardized mix of ranking and rating. Let’s take restaurants. I’m going to rate the last 20 restaurants I’ve been to. Using your equal distribution, that should result in four in each category. So the four restaurants in the lowest category consist of 2 in which I really don’t like the selections on the menu and although I did find something to eat and it was edible, it wasn’t particularly good and I would never go to those restaurants again unless I had no options. The other two however, I didn’t even get as far as ordering and eating. I left when I saw a mouse run across the floor or when I saw an employee scooping coleslaw into a bowl from a big bucket with his bare hands. A rating system that puts those four restaurants in the same category isn’t very useful. Two of them should be lumped in with the places where I like the menu selections better and the food was edible but not particularly good and I might go to that restaurant again if my dining companions liked it. And the same goes for the highest rating - maybe there are one or two restaurants where the menu, the service, the food preparation are far above all the others. I don’t think a system where the “excellent” is lumped in with the merely “very good” is all that useful. Edited: > [@Little\_Nemo](#): > > If I’m rating five restaurants and I feel all five of them belong in the top twenty percent of restaurants, How could you possibly know whether a restaurant is in the top (or bottom) twenty percent of restaurants? Or do you means the top twenty percent of restaurants that you’ve been to? - which is not going to be useful to anyone else. ---
**Author:** ![Little\_Nemo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/little_nemo/32/3120_2.png) [@Little\_Nemo](https://boards.straightdope.com/u/Little_Nemo)\ **Post date:** [December 10, 2022, 1:03am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/29 "2022-12-10T01:03:54Z")
> [@Horatius](#): > > But “this item is complete crap/perfect” is important information, and if your scale doesn’t include those endpoints, it’s inherently incomplete. If you insist on using 1s and 5s equally to 2s through 4s, you obscure the endpoints. “Is this really perfect/crap, or just relatively perfect/crap?” I disagree. I feel my system conveys more information. Let’s say you and I are rating a hundred restaurants and we’re in perfect agreement about their quality. But we use two different systems for assigning ratings. My list might look like this: 1 star - twenty restaurants 2 stars - twenty restaurants 3 stars - twenty restaurants 4 stars- twenty restaurants 5 stars - twenty restaurants Your list might look like this 1 star - two restaurants 2 stars - thirty-two restaurants 3 stars - thirty-two restaurants 4 stars - thirty-two restaurants 5 stars - two restaurants Now I think we’d both agree that there’s no reason to eat at any of the worst restaurants. But I give a list of twenty restaurants that are the worst while your list only identifies two as the worst. Eighteen of the restaurants I rated as among the worst group are given two star ratings in your system. The same is true at the other end. Your system gives the same rating to the third best restaurant in town as it does to the thirtieth best restaurant. But I feel there’s going to be a noticeable difference between the third best and the thirtieth best restaurants. Granted, your ratings system is better at picking out the extremes. Yours is better at picking out the two best restaurants and the two worst restaurants in town. But I feel my ratings system conveys better information about the other ninety-six restaurants we reviewed. ---
**Author:** ![Little\_Nemo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/little_nemo/32/3120_2.png) [@Little\_Nemo](https://boards.straightdope.com/u/Little_Nemo)\ **Post date:** [December 10, 2022, 1:07am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/30 "2022-12-10T01:07:02Z")
> [@Horatius](#): > > Well, at that point we’re just quibbling about what 1s and 5s mean. We seem to be in agreement that, in any real-world collection of items, we could have virtually any distribution. I disagree. I feel that there’s a difference between saying “this movie approaches perfection” and “this movie is in the top fifth of all movies”. I can think of a lot of movies in the latter group that aren’t in the former. ---
**Author:** ![Little\_Nemo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/little_nemo/32/3120_2.png) [@Little\_Nemo](https://boards.straightdope.com/u/Little_Nemo)\ **Post date:** [December 10, 2022, 1:11am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/31 "2022-12-10T01:11:09Z")
> [@doreen](#): > > How could you possibly know whether a restaurant is in the top (or bottom) twenty percent of restaurants? Or do you means the top twenty percent of restaurants that you’ve been to? - which is not going to be useful to anyone else. All ratings systems are arbitrary. If I think _Casablanca_ is the best movie ever made, it doesn’t mean _Casablanca_ is the best movie ever made. It just means I think it is. If you think _Star Wars_ is the best movie ever made, it doesn’t mean my rating is wrong and your rating is right. ---
**Author:** ![doreen](https://avatars.discourse-cdn.com/v4/letter/d/858c86/32.png) [@doreen](https://boards.straightdope.com/u/doreen)\ **Post date:** [December 10, 2022, 1:24am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/32 "2022-12-10T01:24:27Z")
I understand that - what I meant is that you have no basis to think a restaurant is in the top 20% of even a decent sized town because you surely haven’t eaten in all of them , and if you mean that a restaurant is in the top 20% of restaurants you’ve been to, that doesn’t mean very much unless I know most of the restaurants you’ve been to. And actually, I would never think that any movie was the best movie ever made - that’s kind of what I’m trying to get at . At most, it would be the best movie I’ve ever seen , which is very different. ---
**Author:** ![Little\_Nemo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/little_nemo/32/3120_2.png) [@Little\_Nemo](https://boards.straightdope.com/u/Little_Nemo)\ **Post date:** [December 10, 2022, 2:01am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/33 "2022-12-10T02:01:06Z")
This is true. If you’re rating things you have to have some kind of foundation of experience in the things you’re rating. Otherwise you might think your experiencing greatness when all you’re really experiencing is a first time encounter. But I feel once you’ve built up a foundation of experience, you can judge new items based upon past encounters. You may not have eaten at all the restaurants but you’ve eaten at enough restaurants to have a general idea of the spectrum of restaurants. Something can catch you by surprise though. Maybe, for example, you’ve never eaten Thai cuisine before. You go to a Thai restaurant and you love the food. You then rave to people about how great the restaurant is. But later, you’ve eaten at several Thai restaurants and you realize the first one you went to was actually a fairly average one. ---
**Author:** ![thorny\_locust](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/thorny_locust/32/431_2.png) [@thorny\_locust](https://boards.straightdope.com/u/thorny_locust)\ **Post date:** [December 10, 2022, 2:47am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/34 "2022-12-10T02:47:48Z")
> [@Little\_Nemo](#): > > If I restrict the rating of five for things which I judge perfect and the rating of one for things which I judge I’d rather eat broken glass than use (to borrow Horatius’ phrase) then there will be very few things I give a rating of one or five to. What I will effectively have is a system of ratings on a scale of two to four, with pretty much every item receiving one of three ratings. > > I feel a system which used five ratings conveys more information than a system that uses three ratings. Just as a system that rates things on a scale of one to ten gives more information that a binary system that rates things yes or no. The more possible ratings being used in a system, the more detailed the information it conveys. > > So I feel if you have a system which includes five possible ratings and then choose to minimize the use of two of those ratings, you’re handicapping the system. You’d do a better job of communicating by using the full range that you have available. How would it do a better job of communicating if I lied in order to produce such a result? Put it this way, sure, for an item that I usually find some sort of level of satisfactory, I’m rarely going to give a lowest-possible rating. So if you look at a pattern of my answers and see that most things were rated 2 to 4 and only a rare one is rated 1 or 5 – you’ve got more information about those few things that I rated 1 or 5. If instead in an attempt to use all the numbers I’d taken some things I’d usually rate 3 or 4 and rated them 5 instead – you’ve got less information, not more; because you can’t tell the real 5’s from the faked ones, which means that you have no idea whether any of my claimed 5’s are actually 5’s – or for that matter whether any of my claimed numbers are accurate. You really don’t have any information at all, if I’m using that technique. > [@Little\_Nemo](#): > > You’re not following what I’m saying. > > If I’m rating five restaurants and I feel all five of them belong in the top twenty percent of restaurants, then I would give them all a five. And if I felt they all belong in the bottom twenty percent of restaurants, I would give them all a one. Then I’m really not following what you’re saying. Because you also said: > [@Little\_Nemo](#): > > I feel equal distribution makes sense. If you have five ratings and each rating is given to approximately one fifth of the items that seems to make sense to me. If that doesn’t mean that if you were rating five restaurants on a five-number scale you’d give each of them a different number, then what does it mean? > [@doreen](#): > > I would never think that any movie was the best movie ever made - that’s kind of what I’m trying to get at . At most, it would be the best movie I’ve ever seen , which is very different. In addition: I might have seen, say, twenty movies which I thought were really really good – but they were really really good in different ways, and may have all had aspects that were pretty bad. Suppose Movie A had fantastic acting except for the second protagonist, fascinating theme, really well written script, so-so scenery, and really bad historical accuracy. Movie B has fantastic acting by the primary characters, interesting theme, really great scenery except that it doesn’t look like where the movie’s supposedly set, wonderful music, innovative and effective use of camera techniques, and the secondary characters are mostly wooden caricatures. Et cetera. A number rating isn’t going to tell you which is a “better” movie. It isn’t even going to with any sort of accuracy going to tell you which I think is a “better” movie. ---
**Author:** ![Eyebrows\_0f\_Doom](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/eyebrows_0f_doom/32/3119_2.png) [@Eyebrows\_0f\_Doom](https://boards.straightdope.com/u/Eyebrows_0f_Doom)\ **Post date:** [December 10, 2022, 3:19am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/35 "2022-12-10T03:19:57Z")
> [@Little\_Nemo](#): > > Now I think we’d both agree that there’s no reason to eat at any of the worst restaurants. But I give a list of twenty restaurants that are the worst while your list only identifies two as the worst. Eighteen of the restaurants I rated as among the worst group are given two star ratings in your system. Because they _aren’t_ the worst. You are just arbitrarily assigning 20 to each level under some bizarre idea that there are an equal amount of good and bad things. ---
**Author:** ![puzzlegal](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/puzzlegal/32/3827_2.png) [@puzzlegal](https://boards.straightdope.com/u/puzzlegal)\ **Post date:** [December 10, 2022, 3:29am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/36 "2022-12-10T03:29:50Z")
> [@Little\_Nemo](#): > > But I feel my ratings system conveys better information about the other ninety-six restaurants we reviewed. oh boy, do I disagree. Your rating is conflating the 2 restaurants where the food is actually spoiled with 16 restaurants that serve boring sandwiches with too much mayonnaise on stale bread, and two that are actually fine, if unexciting. If I’m stuck picking where to eat out of limited options, I REALLY want to know whether that place is just “I won’t like the food” or “spoiled food”. And if I’m making a special trip for my birthday, I want to know whether it’s a great place or just a very good place. Most real-life distributions are kinda a bell curve. I want to know whether a place is in the tails or in the center. Heck, some real-life distributions, like “does that e-bay vendor deliver as advertised” are highly skewed. I don’t want you to carefully take the ones that wrote answers to my questions in elegant language and set them above the others that were perfectly fine. I want to know whether the vendor delivers on time as advertised. And if 80% of them do, then 80% of them should get the top rating. ---
**Author:** ![Little\_Nemo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/little_nemo/32/3120_2.png) [@Little\_Nemo](https://boards.straightdope.com/u/Little_Nemo)\ **Post date:** [December 10, 2022, 3:52am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/37 "2022-12-10T03:52:10Z")
> [@thorny\_locust](#): > > How would it do a better job of communicating if I lied in order to produce such a result? Do you feel that people who use a different system that yours are lying? ---
**Author:** ![Little\_Nemo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/little_nemo/32/3120_2.png) [@Little\_Nemo](https://boards.straightdope.com/u/Little_Nemo)\ **Post date:** [December 10, 2022, 3:54am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/38 "2022-12-10T03:54:39Z")
> [@thorny\_locust](#): > > So if you look at a pattern of my answers and see that most things were rated 2 to 4 and only a rare one is rated 1 or 5 – you’ve got more information about those few things that I rated 1 or 5. And I have less information about the things that aren’t rated 1 or 5. And as you point out, most of the things aren’t rated 1 or 5. So overall, less information. ---
**Author:** ![Little\_Nemo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/little_nemo/32/3120_2.png) [@Little\_Nemo](https://boards.straightdope.com/u/Little_Nemo)\ **Post date:** [December 10, 2022, 4:03am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/39 "2022-12-10T04:03:16Z")
> [@thorny\_locust](#): > > > [@Little\_Nemo](#): > > > > I feel equal distribution makes sense. If you have five ratings and each rating is given to approximately one fifth of the items that seems to make sense to me. > > If that doesn’t mean that if you were rating five restaurants on a five-number scale you’d give each of them a different number, then what does it mean? You don’t understand. Just because I am placing an approximately equal amount of items in every rating doesn’t mean any random collection of five items will each have a different number. Let’s say I’ve seen a thousand movies in my lifetime. That means around there are around two hundred movies I would rate a five. But that doesn’t mean the next five movies I see will be rated one, two, three, four, and five. If they’re all great movie, they would all be added to the top five-star group. And it they were all really bad, they might all be placed in the bottom one-star group. But in the long run, this sizes of the groups will remain approximately equal. It’s like flipping coins. If you flip a hundred coins, about fifty of them will be heads and about fifty of them will be tails. But that doesn’t mean if you flip two coins, one has to be heads and one has to be tails. ---
**Author:** ![Little\_Nemo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/little_nemo/32/3120_2.png) [@Little\_Nemo](https://boards.straightdope.com/u/Little_Nemo)\ **Post date:** [December 10, 2022, 4:07am UTC](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271/40 "2022-12-10T04:07:13Z")
> [@Eyebrows\_0f\_Doom](#): > > Because they _aren’t_ the worst. You are just arbitrarily assigning 20 to each level under some bizarre idea that there are an equal amount of good and bad things. No, you’re wrong. I’m not placing some arbitrary group of items in the lowest level. I am placing the fifth of the items that I feel are the lowest in quality in the group which I then label as being the lowest. [Previous page](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271.md?page=1) [Next page](https://boards.straightdope.com/t/what-principle-do-you-use-when-ranking-items/976271.md?page=3)