Another Atlantic article with food for thought, delivered by Dash and requesting a five star review…
Are online ratings broken? To some degree they have always been about data panhandling as much as feedback and rewarding both the good provider and silibant ne’er-do-well with a grinder and an axe. I’ve not had the experience of someone asking me to rate whatever more than once or twice…
Excerpt; Atlantic [Ian Bogost] link below
Not to boast, but my feedback is important. So important that, in the past couple of weeks alone, I’ve received a mountain of desperate requests for it.
Amazon, for example, wanted to know if I’d recommend its company based on my Amazon Returns experience. (When the pillow insert I was returning first arrived, the company also asked me to rate my delivery experience.) EGO Power+, the makers of my broken string trimmer, wanted to know if the callback I requested from them yesterday, and missed at 7 a.m. today, had solved my problem—would I complete a survey? When I opened DoorDash to order an acai bowl, the app prodded me to rate Carlos, the dasher who had, days earlier, delivered my Vietnamese noodles, on a five-star scale. An Etsy seller in India from whom I’d purchased a rug sent a fourth message on the app begging me to please rate and review: “It will help my business.” Later, DoorDash also hoped I’d rate the acai joint (separately from the dasher), [but it was complicated since things differed from expectation but I didn’t know who was to blame]…
The issue is that it’s unregulated and they don’t give a rubric so that reviewers can consistenly review things.
So one reviewer might give an item on Amazon one star because the delivery guy left it where it could get wet (not the item’s problem), and someone else might give it a one star “because the robot made me repeat myself while giving it commands.” while another may be like “After five minutes the robot went haywire, set my house on fire, molested my dog, and gave me a wedgie. But for the five minutes it worked well, it worked great. 4 out of 5 stars.”
There’s no consistent way to judge what the reviews actually mean as a result. That’s not even mentioning the issues with fraudulent reviews.
On top of that, some items can’t really even be reviewed in a reasonably fast sort of way. After you buy a washing machine, you won’t really know for a while if it’s actually good.
This has been true since day 1. It’s true of any anonymous and unvetted review system. It’s not that they’re broken; this is a feature of crowdsourced reviews.
My policy is, I never give ratings or answer surveys for anyone unless there’s something specific in it for me. Like a discount on a future purchase. Entering a raffle doesn’t count.
I mostly refuse to give ratings, whether they are pimping themselves for them or not. I don’t pay much attention to amateur, crowd-sourced ratings either, except for buying some kinds of products on Amazon, where it is easy to sample the contents of reviews not just count stars.
It has been ever thus. I can’t see that the quality of ratings has gotten any worse (how could they?) but I agree that providers are getting ever more insistent about them. No thanks, eff off, I have nothing to say to or about you.
At best, there are two ways to usefully use internet reviews. First, you can read the reviews themselves, and see if there’s anything actually pertinent in them to what you are interested in. Second, for items with a lot of reviews, you can do back-of-the-napkin statistical analysis and see how those reviews shake out in terms of how they’re divided between each star category. And if it’s mostly 4-5 stars, read the 1 and 2 star reviews to see WHY they’re panning the item. If it’s mostly moronic stuff dealing with the delivery or customer service, and not the item itself, then you’re probably good. But if it’s actual product concerns, you might want to do more digging or compare it with another highly reviewed version of the same thing. But this only works with thousands of reviews for an item- when there are only like 40 reviews, you can’t really glean much from that.
I still find Amazon reviews useful, particularly when I read the lower ratings. What might bother someone else may not bother me, but if I read reviews demonstrating a recurring problem and think, “Yeah, that would piss me off” then I know to buy something else with a different kind of problem.
Yeah. I keep getting requests for reviews for items I’ve only had a day or two. One of the things I want to know is how well they last! ask me again in a couple of years.
Or don’t; because I very rarely respond to these things at all; in part because they very rarely allow me any way to respond honestly. I don’t know whose fault a problem was and there’s no way to say so; or, ‘the person I finally eventually got on the phone was doing the best they could, but your overall system is broken’ or ‘every individual person seems to be doing the best they can, but your reporting and repair departments aren’t communicating with each other’ – I haven’t seen an online review that lets me say any of those things,
Every couple of months over the last year I’ve gotten an email from the company that rented me a dumpster last year, asking me to rate them on Google. The dumpster was in my driveway for an extra week because their truck broke down, but I didn’t really care; that aside, I really don’t even know how to decide between three stars or four stars or five stars for that kind of service ….
I definitely think internet ratings are broken: there are services that will help businesses game them, users only tend to post reviews under certain circumstances (extremely good or bad experience, for example), or reviews are coerced (customers are told that only 5 star reviews “count”). That said, getting reviews from more than one type of data helps form a more complete picture of the subject of the reviews. Bias exists, but more information helps counter that.
I think they are mostly broken, but not entirely. We had a new restaurant open in our town, we went there once, and because the walls were cement, and it was crowded and popular, it was incredibly noisy. We and a bunch of other people posted that comment on yelp and Facebook, and they responded directly to the Facebook post and ended up putting in some sort of sound baffles.
Also, I buy a fair amount of tools every year online, and it’s helpful to read reviews, not so much the star ratings, rather whether people found the tools useful or not useful for a specific task.
Interesting. I’ve noticed that even when a reputable reviewing company like Wirecutter gives a rave review of a product, there are a significant number of vitriolic one-star reviews on Amazon for the same product.
Wirecutter execs claim the reviews are less heartfelt since it was bought by the NYT(?) and the number of reviews doubled but the staff did not. I never used it enough to be able to notice a difference.
Ratings have always been broken. So is asking people you know if something is or isn’t good. Prior to the internet, what ratings were there? Newspapers would rate a movie, Car and Driver would rate some new cars, Consumer Reports would rate a small subset of popular things. It was all pretty haphazard.
As noted by others, it’s important to understand how ratings work for the place you’re reading them, and use them in ways that provide value despite their shortcomings. Chief among this is the Tolstoy theory (all 5 star ratings are alike, but each 1 star rating is terrible in its own way). Understanding why people were dissatisfied is more useful than knowing that 90% of customers were happy.
I heard on NPR… well, ok it was on Wait, Wait, Don’t Tell Me, that for things like Uber and AirBnb the ratings are basically useless because people give out 5 star reviews way too easily, which makes they worthless as a basis of comparison.
I’ve noticed in reviews on Tripadvisor much of the time a hotel will have a bunch of 4 and 5 star reviews, and then a couple of 1 star reviews complaining that “the staff was rude”. It’s possible that the clerk happened to be having a bad day when that guest arrived, but I always kind of wonder if “rude” really means “They wouldn’t give in to my unreasonable demands.”