Allowing Google to search the SDMB

About This Message Board
1

This question is to my fellow dopers, not TPTB, although, being in power and all, they are free to listen in and participate :wink:

Since Ed Zotti is going to be conversing with the head honcho, this business model for the SDMB is bound to come up in their conversation. Even if we don’t discuss it here, they are very likely to discuss it. AFAICT, it is the most profitable business model the Chicago Reader and whoever owns them could adopt for the SDMB. So we may as well discuss it as a possible future, just to see what the various viewpoints are.

If TPTB were to allow Google to search the SDMB (to be clear this means allowing anyone who respects robots.txt), it would have several immediate benefits. First, the boards would be more available because more hosting resources would be dedicated to serving pages rather than executing searches. Since you would be using Google for your searches, they would complete more quickly and you could execute as many of them as you like, without any limits on the length of the terms and with enhanced syntax. The search engine would never be down.

I consider those to be clear benefits, but there are some other features here that are not clearly good. For example, the biggest risk is that allowing Google to spider the board would put all of this interesting content in their indexes. We would see a huge spike in traffic. All of these folks would be shown ads and some of them would subscribe. There would be an initial slowdown before new hardware could be bought. Additionally, the SDMB would generate a lot of revenue. They would be able to afford expanding the board until it stopped growing. Not all of those search-engine referred members are going to be geniuses. In fact, it’s quite likely that some of them will be downright ignorant. I’m not sure what a manageable genius/ignoramus ratio is, but we’ve got a pretty good balance right now. That’s the worry: Things would get thrown off balance and could change quite a bit. We would be prosperous, but things just wouldn’t be the same, and depending on what you like about this place, they could become worse.

Now I’m not one to ignore the elephant in the room. The biggest concern that anyone will bring up is privacy. I’ve thought about this a bit, and it seems true to me that your privacy is only respected here to the extent that folks willingly respect robots.txt (example of the board’s open door). Anyone who has analyzed the server logs of even a smallish website will tell you that’s not everyone. And the larger your website is, the less true it becomes. Web spiders are cheaper than a dime a dozen and anyone who cares to is free to download every single one of your posts and compile them into a book for private use. After allowing the major search engines to spider the SDMB, your posts might show up in search results. Certainly more often than they do now, which is almost never, although none of us have seen the server logs.

I think this is a good time to discuss the issue. The search engine is currently down because we can’t handle the load because we don’t have enough money to buy enough servers. I know, I know. It’s always been this way. But i’m not sure that it’s ever been this bad. All that needs to be done is to delete the first 3 letters on line 4 of this file and all of those definite woes are traded for a new set of potential woes. So why not?

2

It would possible solve the problem of searching for older posts, though I’d still prefer the build-in search (it just seems more convenient), but it would not bring back the “New posts” feature. I use it on all the message boards I frequent and really miss it.

3

It has been suggested many times. The privacy issue seems to be the sticking point. People mistakenly believe that because Google doesn’t index the SDMB, that no one can search for stuff and find it here.

svrider - there are ways to disable the internal search (which I agree is better than Google if it’s an option) and still use the “New Posts” function. I agree - I miss New Posts link too big time.

4

Actually the New Post search is a minimal impact on the Dope. It is the fastest search as it only looks at threads from the last day. (I think) I notice it is very quick compared to any advance search of over a day back.

5

I have occasionally been able to find certain posts I was looking for on Google by entering keywords in the appropriate spots in the “Advanced Search” fields.

What I miss most about the Dope’s search function being down is the ability to bring up all of a poster’s previous posts from their user profile or by clicking on their name.

6

It’s not only privacy, it’s also the fact that it’s one of the features we’ve paid for. One that many people felt was worth their $7.95 or their $15.

Be like paying for a ticket at a baseball game only to find that everyone is allowed in for free after the third inning.

7

People really come here and expect privacy?

This isn’t sending messages to your doctor or the bank over an SSL connection complete with https:// and the little padlock icon lighting up.

This is a public message board that’s freely available to any person that can figure out how to get here. I certainly don’t come here with any expectations of privacy. I’ve always been in favor of the idea to let Google crawl the board, which would then let us search like meerkats on crack whenever and as often as we wanted, and we wouln’t have the artificial 4-letter minimum. And, we wouldn’t be consuming CPU cycles.

As someone else said, if people can find things here through Google, some percentage of them will look around and think “Hey, this place is neat!” and buy a membership. Right now, we’re comparatively difficult to find - other than bumbling in off of the Mailbag, how do new people find us?

8

I don’t think it’s privacy so much as that searching is touted as one of the benefits of subscribing. If the archive becomes searchable via Google, the admins lose a major selling point and the subscribers lose something that makes them feel special.

9

It’s a fair point, but I think that particular benefit is far overshadowed by the ability to actually ask questions of your own (at least for longer than 30 days.) We’ve got a vast repository of information in the archived posts in GQ and GD, and I could see SDMB threads becoming highly-ranked search results; if people are drawn to the board by that and later realize that they want to join the community, so much the better.