Yelp is Yuseless (Filtering)

The BBQ Pit
41

I can bring this all down to a very specific case. Judge ye as ye will:

My little town is notably short of places to eat that (1) don’t serve pizza and (2) don’t have paper advertising placemats. You have to drive a town or two over to get much selection beyond that, and into the city to have any real choice. (I enjoy pizza, and diner food, but I’ve eaten in some of the best restaurants across half the globe and love to do so.)

So a high end, name-chef place opens in town. We go, with great glee, a month into their unannounced opening.

Meh. Two good entrees out of five, very questionable ambience and service quality, everything else very iffy, and incredibly slow service (over 90 minutes to entrees, with the table empty for 20 minutes at a time). About $50 a person with no significant extras.

We chalk it up to opening bumps, as the proprietors have a good name elsewhere. We go back about six months into operation.

Two good entrees among four. My side dishes were ice cold and the vegetables were like sawing wood. Waitstaff that seemed to have no clue what the job involved except for hanging around bothering people. Deadly slow service (well over 90 minutes to entrees). Very crowded; every person that went in and out of our room bumped into someone at our table, and service trays passed over our heads. The lights flickered like a prison movie the whole time. Half the food was on the plates when they cleared the table; no questions asked. Drinks were collected a half hour before that, no offer of refills. A little over $50 per person, no significant extras.

I wrote this in a long, detailed, dispassionate review, gave two stars. I think it was my sixth or seventh Yelp review in about six months.

It disappeared. This is when I discovered filtering, and found this place, in addition to its three four- and five-star public reviews, one of which screams shill, had FOURTEEN filtered reviews, three of which were good and eleven of which essentially reported the same thing I did - except that some reported that if you complained, the owner/chef would come out and berate you publicly, insinuating that you didn’t know shit about fine dining and should try Olive Garden next time. I did not detect a single slam or evident shill among them; I in fact knew who several of the reviewers were (small town, remember).

The NY Times reviewed them, based on the chef name, and while they found good things to say, it was just snarky enough that I can’t see the review having drawn anyone there. The endlessly flickering lights particularly bemused them.

I gave the restaurant until after the holidays, about an eight month run.

They closed in October, a six month go.

Anyone using Yelp at face value would have thought they were going to a fabulous, out of the way gem. IIRC, at least a couple of the reviews started with that comment and went downhill fast. At the cost, I think people deserved better warning and more accurate reviews and ratings. This was as far from a four- or five-star joint, at these prices, as I’ve ever been in.

So yes, I think it matters that Yelp’s review system is so skewed.

42

Do you mind pointing to examples please?

I work in spam and fraud detection. Not specifically with reviews, but I’ve worked with people who do. This is a non-trivial problem. As spinky says, there’s a lot of money to be made if you’ve solved it.

43

Also, bad eateries go out of business, taking their 1&2 star reviews with them.

44

Let me back this all up: I didn’t make my point clear and trying to reply to somewhat off-center questions just makes it more confused.

I didn’t in any way mean that I was capable of crafting a standalone AI that would handle the problem autonomously. (In framing that, another part of the Yelp problem outlined itself - get there in a moment.) You’re right in that trying to create such a beast, with anything like the ability of spam and virus detectors, would be something to get one’s name on the Nobel short list.

I don’t think there is ANY achievable solution, even partial, that doesn’t involve a human component. Automated filtering can only apply a set of judgments and then kick out some percentage of indeterminates for human review. I don’t see writing (the meta-instructions for) something of that scale as something beyond my capabilities; I can think of 20 or 30 criteria that would cover a significant part of the spectrum. That would need an evolving analysis scheme to adapt to changing elements, but again I don’t see it as an overwhelming task… IF the system has an option to kick certain gray-area instances out for human eyes. You don’t need to design or program for those last nines or next SD if you have that, and the task gets OOM easier.

(Also consider that most spam, virus and phishing comes from anonymous or spoofed sources; simply requiring a validated registration for reviews, as nearly all sites do, cuts that problem immensely. Your evaluation is of a registered user base, not a random universe.)

To refine my comments about Yelp, I think that’s precisely what they’re trying to do, make the review analysis 100% AI. Since that can’t be done, it’s flawed. Since they don’t appear to have, or be willing to commit, the resources to such a task, it’s flawed-er. That seems to outline exactly the kind of peculiarly erratic results we’re seeing.

I maintain that Yelp is yuseless and what’s more, they know it - otherwise they would not be so coyly defensive of a system that doesn’t hold up to real-world evaluation. They’re trying to do an immense job on a limited budget - which would be fine, and understandable, but THIS IS THEIR ENTIRE PURPOSE. They aren’t Amazon or B&H Photo or NewEgg or anyone else primarily selling things and providing a review space; Yelp exists for no purpose but to host and enable this review/rating service. And they’re cutting big corners on it, making it an exercise in uselessness. What’s worst here is that probably 7 out of 10 users or more have no idea that the reviews are all but random and the ratings meaningless. I don’t find such skewage on TripAdvisor, for example, so I know a better job can be done… and I believe that even with my comparatively modest skills, I could do better than they do now… mainly because I’d rely on a human component to backstop the AI.

Yelp sucks… because they are choosing to suck.

45

Bad eateries go out of business, one would hope, regardless of their ratings. See my post above; from the Yelp listing and reviews, the restaurant was a jewel worth a 50-mile drive.

I think, too, that a lot of those who are defending Yelp here or saying they find the system adequate are playing the game with white chips; if a $10 lunch in a pizza place is bad despite fabulous reviews, you haven’t lost much. You don’t go back and assuming you don’t get food poisoning, it’s so-what.

On the other hand, if you’re going to choose more expensive restaurants, make plans for a special evening or select businesses with expensive services (tranny repair? body work? dentist?) based on Yelp, you could get ripped off for a painful amount, all under Yelp’s benign and placid smile.

46

I think you are vastly underestimating the scale and nature of the problem.

It’s certainly true that requiring registration or using a CAPTCHA will cut the bulk of it on a low-profile target. But on a high value target like Yelp, the bad guys long ago switched to methods that are not affected by those things. They use humans. They bribe established reviewers. They compromise accounts. They have years of experience testing around the edges to figure out what works and what does not. And there are many people doing these things in parallel.

It’s trivially easy to come up with ideas for stopping spam and scams that covers the cases you know about. But as you scale up you will find that the things you don’t know about outnumber those you thought you did when you started.

Using humans doesn’t magically fix that. Scammers are social engineers. They know how to manipulate moderators, how to fool people into giving them the benefit of the doubt.

47

Didn’t Zagat get caught, years ago-releasing “reviews” that were written before the restaurants opened? I can see why you want an honest rating system-a dinner out is a big investment in money…I want to be assured that I’m getting value for my money.

48

Indeed. How often do you just pick a restaurant at random, excpet on the road? You pick a restaurant as it’s a favorite (so of course you write a good review) or that a friend has recommended it, or you read a good review. Rarely do we go to a restaurant where your friend or reviewer has said “stay away from this place”.

49

I’m unsure of your point. That’s (in theory) the entire reason for reviews and ratings; if they are dishonest or skewed, you can’t use them to choose or avoid a new and unknown place to try. As in my long example, I don’t doubt that some significant number of people elected to try the restaurant based on the skewed public selection (4-1/2 stars, three rave reviews)… and were sorely disappointed.

I have learned to read through a selection of user reviews on pretty much anything - product, restaurant, hotel, etc. - and apply my own filtering to come up with a reasonable judgement. I haven’t been fooled too many times. But Yelp is trying too hard, with ineffective tools, to do that job for everyone, and I’ll concede tellyworth’s point that it’s a hell of a lot harder than it might seem. Without having a less aggressively “fixed” list of reviews to work with, Yelp’s listings are good as a name and address listing, and not much more - their skew factor, however well-intended or not, simply makes hash of the reviews and ratings. You’d think they’d have figured this out by now and backed away to a more sensible, if less “padded edge” model.

But, of course, if 95% of Yelp’s users are unaware of the filtering, there wouldn’t be much reason for them to change.

Don’t buy Yelp stock, guys and gals. I’d predict its ratings to have a sudden fall, someday.

50

Geez, their automatic system doesn’t like one of your reviews, and you now have a hate on for them.

51

I’ve read this whole thread, to my misfortune, and the gist I got is, “I don’t wanna play by Yelp’s rules but I want them to treat my reviews as if I did.”

What did you pay for Yelp?

52

Yes. Now wait a minute. I just recognized the OP, he was the same dude complaining about Amazon filtering his reviews:
http://boards.straightdope.com/sdmb/showthread.php?t=674812&highlight=amazon

Hmm, there must be some giant conspiracy against him. Or- he just includes words in his review which are forbidden and won’t fess up about it.

53

Fine, I’m done.

My discovery of Yelp’s filtering process, and further research into it by looking at 30 or 40 businesses I was familiar with, was sparked by having my own (genuine and accurate) review filtered. I don’t much care that the review was filtered - nor that Amazon rejected a review for equally puzzling reasons - but when the pattern is chaotic and meaningless, I though it worth pointing out.

Since Amazon accepted my second review on the same product, with essentially the same content, I have no problem with them. I can see how a bot might count up too many check marks against even a sincere review from a long-time reviewer and Amazon merchant, but the system is adaptable enough not to be capricious.

Yelp? I’ll never use Yelp again. If you want to chalk it up to hurt feelings, be my guest. If you can’t see the problem with their system, use it in all good health. Use it to choose a top-drawer restaurant for your next pivotal date with Mr/Mrs OhMyGod. Much luck to you.