I punched in my user name on Google, and was surprised that none of my SDMB threads came up. I checked a few other user names of my fellow dopers, but none of the results linked to the SDMB.
So, I’m wondering…does Google deliberately exclude message boards from its index? Or did the Straight Dope specifically request to not be spidered by Google?
I’m actually a bit relieved…if & when someone "googles" me, I’d rather not have every single one of my opinions & views available to them at their fingertips…
Seriously though, I very well could be wrong, perhaps the SDMBis indexed on Google. If so, can someone give me a search term that will bring up a particular thread? The Straight Dope home page is linked, as is the SDMB front page, but so far, I haven’t been able to find any indexed threads…
I always thought that the MB threads aren’t indexed on Google because they’re not static; they’re dynamic, e.g. only created when someone issues a request to view the thread. But let me check on that.
The server generates the webpage on the fly, depending on what the threadID is. In order for Google to index the pages, they’d have to know the parameter name ‘threadid’.
I think that Google just finds stuff by following links. So SDMB pages that someone has linked to from elsewhere might get included, but in order to get the whole thing, Google would have to keep updating its database to get new threads and updates of older threads.
Well, I’ve been startled a couple of times to have the very thread I was doing Google research for (in another window) come up in the list of Google results. I don’t remember which threads they were, but both times I was Googling for a particular (and unusual) phrase that also happened to occur in the thread (Like, oh, say, “mammoths consume lime jello with no digestive upset”) and saw the thread itself pop up on Google Hit #1 or #2.
And, Google couldn’t specifically exclude message boards from its searching, because a goodly number of the hits you’ll get for hobbyist stuff, like for pets or home repair, and also for medical conditions, tend to be posts on message boards.
But message boards are wise to exclude themselves from Google, and probably do via a “robots exclusion”. When you think of the server issues we have here, do we really want Google, Altavista, etc. searching the entirety of the database every couple of weeks or so?
I hope they’re excluded - I don’t like the idea of this message board being indexed on Google. It’s one thing to share opinions with people who come here to view them, a very different one to share with the world at large.
CnoteChris, that’s only the start page. The links lead to our ‘real’ board. Thought we wouldn’t notice, eh?
The only entry in straightdope.com/robots.txt leads to “Hey! You’re not supposed to be rooting around in here!” :o Crawlers should be free to go all the way down from the start page. Link is link, dynamically created page or not.
But could we purposely once a day send the results of a search-for-new-threads to Google and let their healthy hamsters do the dirty search work for us? Do they offer such a service? All we had to do is write a script and a Google search template on our site containing ‘site:boards.straightd…’ Worth the effort? Discussed before?
You’re right, I have to admit it - I actually thought people would head over to that site and actually check out more than one lone link before they came back and bitched at me. If they had, they might notice that not all links over there revert back to the current board. In fact, if they clicked over there enough times, they might even notice that appears to be a problem with the browser, and not necessarily that site. Notice when it bumps you back here, to the current site, that the prefix ‘web.archive.org’ is lost? That’s gotta be, and mean, something. What it is, I don’t know - I’m not hip on http.
Be patient, my man, and check it out. When it all comes together and actually works, you’ll find some interesting things from back then. Take a look at a early Napster rant circa May 5, 2000 - Napster!. Kind of a fascinating discussion, when you look back on it now.
I was about to give you some off-the-cuff answer, Biggirl, on how it’s a big government plot, but then I thought better about it.
How I found it was from a newsletter I get that had an article on it a few months back. Why they do it is anyone’s guess, but they do. The organization is the Internet Archive. Type in a website in the ‘Way Back Machine’ space and it’ll give you a list of it’s archived files from any number of past dates.
I found it so interesting at the time that I immediately came here and posted about it, only to find out a short time later that so did about ten other posters. Those posts came and went and I haven’t seen or heard much of that site discussed since.
Too bad, because it can be cool as hell finding old stuff from the net’s past.
CnoteChris, Well, my mouse did hover over more than one link. But that site runs a java script to replace the links. Since I - say - “don’t like” surfing around with JavaScript turned on, I didn’t notice that I wasn’t shown the same links as you. Made worse by my vague memory of a thread where such a link confusion stunt was actually pulled as a joke. Sorry.
