Let's all watch Google crawl the Dope

Is this a precursor to shifting all searching via Google?

Because if that happens, I fear we will no longer have as much control over what kinds of results we get. If I vanity search myself, I can restrict it to time periods, and forums, and then the results appear in chronological order.

But Google does not have that kind of set up, so I’ll get results from 5 years ago, 9 years ago, 3 weeks ago, and 25 minutes ago, in a random order.

This is not what I want at all.

How did that work out for you? :slight_smile:

I searched for my username and the posts appear to be in date order, from newer to older.

I’ll just be happy to get results, and not an error message. And if I want to alter my search, not wait 5 frickin minutes to try again.

False.

If you want to do a Cafe Society search, then you include “cafe society” in your search criteria. If you want to limit it to the last month, then you go into advanced search and limit it to the last month.

Mar 13 - 3,500
Mar 14 - 4,700
Mar 15 - 8,100
Mar 16 - 10,000
Mar 17 - 12,100
Mar 18 - 15,000

At this rate it will take Google half a year to crawl all 500,000 threads. I suspect we will see some major spikes in Google’s crawling rate once it “realizes” how much original content is here.

Wow, I got a little bit of a pile-on there.

I use the Google Toolbar exclusively, so had no idea there was an Advanced Search for Google.

Just a post to keep track of this thread.

Mar 20 - 18,000
Mar 21 - 20,800

Ummm… let me understand this. The SDMB administration has been trying for years to find a viable way of making this a self supporting enterprise either through subscriptions or delivering eyeballs to advertisers, and we’ve been deliberately preventing access to the SDMB content until approx 10 days ago by the largest, most powerful search engine in the world.

Is that understanding correct?

One of the problems with fighting ignorance for 36 years is becoming old and not realizing that new technologies can solve all your problems.

I’m not in the know, but I believe that the rational for not allowing Google’s content bot to search until now was the fact that the servers we had prior to the last upgrade were so limited that the Board performance would suffer terribly. Just what I remember.

Thanks a lot – using that, I just figured out something completely new to me, a way to search threads in such a way as to reveal who posted how often to them: enter the url [noparse]http://boards.straightdope.com/sdmb/misc.php?do=whoposted&t=[/noparse] into your address bar, and put a thread number behind the ‘t=’. Doing so for this thread brings you here. Clicking again on the little number behind the username lists only the posts by that poster within that thread.

I’ll probably never use this again, and likely everybody knew this except for me, but I think it’s pretty nifty. Are there any other tricks like this that have so far eluded me?

Google will not crawl through the links on a page where it’s not welcome, but it will still be aware of the existence of things linked from pages that do welcome it. So the classic mega-threads like The Horror of Blimps and If LotR Had Been Written By Someone Else!?, which got linked in many places outside the Dope, were already in Google’s index, though it didn’t know anything about them other than what it got from the linking pages.

Google has always indexed the Straight Dope columns (as opposed to the board), and due to the esoteric nature of the questions Cecil covers, they usually end up near the top of searches. You probably would have gotten the same result if you had searched years ago. Which question was it, incidentally?