BoardReader is profiting off of SDMB content when it could be the Reader. One of these days they are going to realize this and let Google in 
I’ve pointed this out several times recently. The better page to use is the advanced search page where you can put “straightdope.com” in the domain field.
Some people have said that it doesn’t find some threads, but every time I’ve needed to search for something, it has turned up on BoardReader. The only disadvantage I see is that it takes a week or so for BoardReader to find out about new content, meaning that very new threads won’t show up.
I don’t see it as a lopsided profit issue. Don’t we get more traffic and potential subscribers if more people can find our pearls of wisdom via BoardReader? Why limit the means by which people can find us?
Because BoardReader is violating the robots.txt honor system. They are specifically asked not to crawl the boards, and they choose to do so anyway. And they are profiting off of it.
Be that as it may, this is a useful stopgap. And I’m not going to overly concern myself with their ethics while I’m being deprived of one of the major benefits of subscription.
Hmm - they do claim to respect robots.txt, but judging by posts on their forum it looks like they only skip sites that explicitly forbid them, i.e. they ignore the wildcard entry that our site uses. It’s also possible that their spider isn’t bright enough to get the correct file. They do seem to respond to email requests not to spider, at any rate.
I don’t really see the problem, regardless. It’s a pain in the ass to get readable threads out of their site without coming here anyway, and they’re the only practical way to find anything on this board at the moment, so getting us removed from their index really would be cutting off our nose to spite our face.
Oh No!
What exactly is the problem? We certainly can’t profit from it at the moment. In fact the search feature has become a liability; that’s why it has been disabled. This is a temporary outsourcing of the search feature, that’s all. As long as the crawlers are reasonably unobtrusive, seems like a win/win.
You would prefer to have a third party profit off of providing a substandard search engine service to board members who paid the Reader for that very thing rather than having the Reader make that money so that they can provide a high quality service over the long term? In addition to not honoring our robots.txt, which all major search engines and responsible folks do, we have no idea as to how optimal BoardReader’s spider is. For all we know it download’s the SDMB’s full archives once a week, thus slowing things down even further. Such a “stopgap” seems harmful. If anything proactive measures should be taken to block their spider’s IP.
It works, but is a pita imho.
I just want the regular search back.
Thanks, Patty. I found the Mariana sauce in the pan on the first search page.