# Let's all watch Google crawl the Dope **URL:** \ **Category:** Miscellaneous and Personal Stuff I Must Share\ **Created:** [March 15, 2009, 11:39pm UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566 "2009-03-15T23:39:01Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [March 15, 2009, 11:39pm UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/1 "2009-03-15T23:39:01Z")
Did you hear that Jerry is letting Google crawl the boards? It’s true, have a look at [robots.txt](http://boards.straightdope.com/robots.txt). I’ve been watching and Google is catching on to this. The query is: [http://www.google.com/search?q=site:boards.straightdope.com](http://www.google.com/search?q=site:boards.straightdope.com) [ul] [li] Mar 13 - 3,500 pages[/li][li] Mar 14 - 4,700 pages[/li][li] Mar 15 - 8,100 pages[/li][/ul] Does anyone else think that this place is about to start getting a _lot_ more traffic? A lot more memberships, a lot more features, and a lot more ignorant? Or is our backbone of ignorance fighting skill enough to challenge the unwashed masses of the world, which we have never before confronted? Or will Google’s access be shut off when the servers buckle under the load? ---
**Author:** ![samclem](https://avatars.discourse-cdn.com/v4/letter/s/a9a28c/32.png) [@samclem](https://boards.straightdope.com/u/samclem)\ **Post date:** [March 16, 2009, 12:14am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/2 "2009-03-16T00:14:46Z")
> [@alterego](#): > > Does anyone else think that this place is about to start getting a _lot_ more traffic? A lot more memberships, a lot more features, and a lot more ignorant? Or is our backbone of ignorance fighting skill enough to challenge the unwashed masses of the world, which we have never before confronted? Or will Google’s access be shut off when the servers buckle under the load? I really think we’re up to it. “Bring it on” to quote someone. 🙂 ---
**Author:** ![WordMan](https://avatars.discourse-cdn.com/v4/letter/w/e8c25b/32.png) [@WordMan](https://boards.straightdope.com/u/WordMan)\ **Post date:** [March 16, 2009, 12:22am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/3 "2009-03-16T00:22:09Z")
So we weren’t scanned by Google before? And now we’re getting more traffic because, as Google “assimilates” us (sorry) we turn up on more search results so more folks find us?? ---
**Author:** ![Der\_Trihs](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/der_trihs/32/233_2.png) [@Der\_Trihs](https://boards.straightdope.com/u/Der_Trihs)\ **Post date:** [March 16, 2009, 12:23am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/4 "2009-03-16T00:23:50Z")
> [@alterego](#): > > Did you hear that Jerry is letting Google crawl the boards? It’s true, have a look at [robots.txt](http://boards.straightdope.com/robots.txt). How do you interpet this ? I got : > [@](#): > > User-agent: \* > Disallow: / > > User-agent: Mediapartners-Google\* > Disallow: > > User-agent: Googlebot\* > Disallow: Do those “disallowed” mean it’s been shut off ? Or is it the blankness after ‘Disallow’ that means Google can scan the Dope ? I’m clueless about this . . . ---
**Author:** ![Reply](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/reply/32/15952_2.png) [@Reply](https://boards.straightdope.com/u/Reply)\ **Post date:** [March 16, 2009, 12:28am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/5 "2009-03-16T00:28:56Z")
Holy shit! Scary crap. The invasion begins… ---
**Author:** ![Sage\_Rat](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/sage_rat/32/399_2.png) [@Sage\_Rat](https://boards.straightdope.com/u/Sage_Rat)\ **Post date:** [March 16, 2009, 12:33am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/6 "2009-03-16T00:33:40Z")
> [@Der\_Trihs](#): > > How do you interpet this ? I got : > > Do those “disallowed” mean it’s been shut off ? Or is it the blankness after ‘Disallow’ that means Google can scan the Dope ? I’m clueless about this . . . It’s saying that unless you’re Google to go away. If you are Google, then you aren’t disallowed from anywhere. ---
**Author:** ![Der\_Trihs](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/der_trihs/32/233_2.png) [@Der\_Trihs](https://boards.straightdope.com/u/Der_Trihs)\ **Post date:** [March 16, 2009, 12:36am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/7 "2009-03-16T00:36:38Z")
> [@Sage\_Rat](#): > > It’s saying that unless you’re Google to go away. If you are Google, then you aren’t disallowed from anywhere. Thanks. ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [March 16, 2009, 12:39am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/8 "2009-03-16T00:39:06Z")
> [@WordMan](#): > > So we weren’t scanned by Google before? And now we’re getting more traffic because, as Google “assimilates” us (sorry) we turn up on more search results so more folks find us?? That’s the idea. ---
**Author:** ![WordMan](https://avatars.discourse-cdn.com/v4/letter/w/e8c25b/32.png) [@WordMan](https://boards.straightdope.com/u/WordMan)\ **Post date:** [March 16, 2009, 12:50am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/9 "2009-03-16T00:50:00Z")
> [@alterego](#): > > That’s the idea. Well fight my ignorance! ---
**Author:** ![Revtim](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/revtim/32/1042_2.png) [@Revtim](https://boards.straightdope.com/u/Revtim)\ **Post date:** [March 16, 2009, 1:51am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/10 "2009-03-16T01:51:47Z")
The best part is that we can use Google ourselves instead of the god-forsaken search board engine, once it’s all indexed. ---
**Author:** ![Robot\_Arm](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/robot_arm/32/18280_2.png) [@Robot\_Arm](https://boards.straightdope.com/u/Robot_Arm)\ **Post date:** [March 16, 2009, 1:56am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/11 "2009-03-16T01:56:19Z")
> [@samclem](#): > > I really think we’re up to it. > “Bring it on” to quote someone. 🙂 Are you sure about that? I just checked, and the fourth result is “the horror of blimps”. ---
**Author:** ![AskNott](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/asknott/32/5790_2.png) [@AskNott](https://boards.straightdope.com/u/AskNott)\ **Post date:** [March 16, 2009, 3:23am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/12 "2009-03-16T03:23:30Z")
Let’s not pretend Gooooooogle has not been aware of us up til now. I googed myself a year ago, and the first two hits were from here. ---
**Author:** ![Boyo\_Jim](https://avatars.discourse-cdn.com/v4/letter/b/87869e/32.png) [@Boyo\_Jim](https://boards.straightdope.com/u/Boyo_Jim)\ **Post date:** [March 16, 2009, 3:31am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/13 "2009-03-16T03:31:15Z")
Jeez, I used to be concerned about letting people here know my real name. Now, I need to worry about the people I know in the RW knowing my username here. If they Google “Boyo Jim”, not only can they find my batshit crazy posts, but even pictures of me. :eek: Ah, WTF… 😉 ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [March 16, 2009, 4:43am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/14 "2009-03-16T04:43:42Z")
> [@AskNott](#): > > Let’s not pretend Gooooooogle has not been aware of us up til now. I googed myself a year ago, and the first two hits were from here. Google doesn’t index all the sites it’s aware of. It does obey robots.txt ---
**Author:** ![Princhester](https://avatars.discourse-cdn.com/v4/letter/p/3e96dc/32.png) [@Princhester](https://boards.straightdope.com/u/Princhester)\ **Post date:** [March 16, 2009, 4:44am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/15 "2009-03-16T04:44:58Z")
Isn’t there going to be an Invasion of Zombie Threads? People will google something and land up in some old thread about pan frying gophers on a conveyor belt in the Marianas Trench, sign up and start posting to the old thread. I think a mass lock-off is in order ---
**Author:** ![Spectre\_of\_Pithecanthropus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/spectre_of_pithecanthropus/32/12343_2.png) [@Spectre\_of\_Pithecanthropus](https://boards.straightdope.com/u/Spectre_of_Pithecanthropus)\ **Post date:** [March 16, 2009, 5:43am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/16 "2009-03-16T05:43:35Z")
> [@Princhester](#): > > Isn’t there going to be an Invasion of Zombie Threads? People will google something and land up in some old thread about _ **pan frying gophers on a conveyor belt in the Marianas Trench** _, sign up and start posting to the old thread. I think a mass lock-off is in order You left out…“while traveling by blimp”… My comment generally: [“Newman” voice (from “Seinfeld”)] JERRY! [/"Newman v.] ---
**Author:** ![Spectre\_of\_Pithecanthropus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/spectre_of_pithecanthropus/32/12343_2.png) [@Spectre\_of\_Pithecanthropus](https://boards.straightdope.com/u/Spectre_of_Pithecanthropus)\ **Post date:** [March 16, 2009, 5:47am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/17 "2009-03-16T05:47:00Z")
Just googled for my recent MPSIMS OP on the “One word–Plastics” dialog from _The Graduate_, and there it is. I don’t know whether to e happy or sorry. ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [March 16, 2009, 6:18am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/18 "2009-03-16T06:18:03Z")
> [@Spectre\_of\_Pithecanthropus](#): > > Just googled for my recent MPSIMS OP on the “One word–Plastics” dialog from _The Graduate_, and there it is. I don’t know whether to e happy or sorry. I just Googled for my nickname and found this thread. Google even displays that there were 10 posts and 6 authors when it came. It looks like it found it almost as soon as I posted it. I just created a Google Custom Search Engine for the Dope. It only returns results from the boards. [Here it is](http://www.google.com/coop/cse?cx=001223136417691641822:x0mhldiy7ko). ---
**Author:** ![DMark](https://avatars.discourse-cdn.com/v4/letter/d/e68b1a/32.png) [@DMark](https://boards.straightdope.com/u/DMark)\ **Post date:** [March 16, 2009, 8:09am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/19 "2009-03-16T08:09:36Z")
My very first, and only, question to Cecil (with his answer) is now number THREE on a Google search with the same topic! Amazing! Especially considering the question was asked in 1997!!! Plus, I am kind of amazed that DMark (the German currency from after WWII until the conversion to Euro) turns up my name on Google on page 5! ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [March 16, 2009, 8:11am UTC](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566/20 "2009-03-16T08:11:30Z")
> [@DMark](#): > > My very first, and only, question to Cecil (with his answer) is now number THREE on a Google search with the same topic! Amazing! Especially considering the question was asked in 1997!!! > Plus, I am kind of amazed that DMark (the German currency from after WWII until the conversion to Euro) turns up my name on Google on page 5! I’m tellin’ ya’ - there’s gonna’ be a flood. [Next page](https://boards.straightdope.com/t/lets-all-watch-google-crawl-the-dope/489566.md?page=2)