Dope Search

This implements most of the stuff in my howto, except for the julian date thing. I almost did that. Maybe next time.**

index.html**



<frameset rows="30%,70%">
<frame src="search.html" frameborder="0">
<frame name="google" src="http://www.google.com/search?q=site:boards.straightdope.com" frameborder="0">
</frameset>


search.html



<html>
  <script>
var forum, recency, title, query;
var url = "http://www.google.com/search?q=site:boards.straightdope.com";

function UpdateQuery() query = document.getElementsByName("query")[0].value;
function UpdateTitle() title = document.getElementsByName("title")[0].value;

function SearchForum() {
  forum = document.getElementById("forum");
  forum = forum.options[forum.selectedIndex].value;
}

function SearchRecency() {
 recency = document.getElementById("recency");
 recency = recency.options[recency.selectedIndex].value; 
}

function DopeSearch() {
  var username = document.search.username.value;
  if (username) url += ' %2B"Find all posts by ' + username + '"';
  if (forum) url += ' %2B"' + forum + '"';
  if (query) url += ' ' + query;
  if (title) url += ' intitle:' + title;
  if (recency) url += recency;
  parent.google.location = url;
}
  </script>
  <center>
    <h1>Dope Search</h1>
    <form onsubmit="DopeSearch()" name="search">
      Forum: 
      <select id="forum" onchange="SearchForum()">
    <option value="All forums" selected="selected">All forums</option>
    <option value="About This Message Board">ATMB</option>
    <option value="Comments on Cecils Columns">CCC</option>
    <option value="General Questions">GQ</option>
    <option value="Great Debates">GD</option>
    <option value="Cafe Society">CS</option>
    <option value="The Game Room">TGR</option>
    <option value="In My Humble Opinion">IMHO</option>
    <option value="Mundane Pointless Stuff I Must Share">MPSIMS</option>
    <option value="The BBQ Pit">The Pit</option>
      </select>
      Recency: 
      <select id="recency" onchange="SearchRecency()">
        <option value="All times" selected="selected">All time</option>
    <option value="&tbs=qdr:d">Last year</option>
    <option value="&tbs=qdr:m">Last month</option>
    <option value="&tbs=qdr:w">Last week</option>
    <option value="&tbs=qdr:d">Last day</option>
      </select>
      User:
      <input type="text" name="username" size="10%">
      <br/><br/>
      Title contains:
      <input type="text" size="50%" name="title" onchange="UpdateTitle()">
      <br/><br/>
      Query:
      <input type="text" size="50%" name="query" onchange="UpdateQuery()">
      <input type="submit" value="Search">
    </form>
  </center>
</html>


Thanks. I assume this is for a webpage to be hosted somewhere. (I am a mainframe guy.)

Would this be the appropriate place to ask dumb questions about your post on how to search the Dope using Google? Assuming the answer to be Yes -

If I do (for instance) a vanity search for all posts that mention me between 2008-2009 like this

site:boards.straightdope.com Shodan daterange:2454466-2454832

it comes up with three results. I have written more posts than that during that period, and I almost always sign my posts, so there must be something I am doing wrong. What is it?

Also, is there any way to sort the result set (or whatever the correct term is)?

Thanks in advance.

Regards,
Shodan

It looks like he wants it hosted here. It’s a frameset, with a fairly generous portion (not quite a third) devoted to getting all your search parameters, including forum. A bit more than two-thirds is Google itself — the result set, upon submitting the criteria. No sorting specified. Perhaps Google takes care of that, I don’t know. There is a “query” field though, which might allow you to specify things like that. Not sure. The whole thing is submitted to Google via a form “get” (apparently), just using an ordinary query string.

Google is not a great option to completely replace the stock search because only 10-15% of the board is spidered.

Is that why only three results came up for my search?

Thanks for your response.

Regards,
Shodan

Well, it’s hard to say in your case. My understanding is the break in spidering is because the SDMB went to a new format for thread and post URLs and Google can’t seem to find the older posts. But you’re searching recent posts-- ones that I believe should have been covered by Google’s spider without much trouble.

Another problem with Google is they’ve got so many nodes that search results will be inconsistent. What you come up with in a search is not necessarily going to be what someone else comes up with. At least with the stock search there is some consistency in results.

Something is definitely fishy with the date range search. The “entire” year of 2008-09 returns about 240 results.

I uploaded the OP’s code to one of our servers. There’s a missing opening bracket somewhere, and I just don’t have time to dog it down. Maybe the OP could have a look, post the new code, and I’ll give it another try.

You’re doing it right. For some reason Google didn’t get all the board contents. If we can convince TPTB to submit a Google Sitemap of the board contents then you would get all the results.

Google does all the sorting. The best you can do to bias the results is add more keywords.

I don’t think Google has many posts from that time period, for whatever strange reason.

I just copied/pasted pmy posted code onto a new computer and it worked!

Is there any way you can make this work locally? Maybe just give the search string for one to use in google?

This new version:
[ul]
[li]Shows lengthy text snippets[/li][li]Defaults to new posts in the last day[/li][li]Allows you to search a date range in M/D/YYYY format[/li][li]Allows you to sort by either relevance or date[/li][/ul]

index.html


<frameset rows="30%,70%">
<frame src="search.html" frameborder="0">
<frame name="google" src="http://www.google.com/search?q=site:boards.straightdope.com&tbs=frm:1,sts:1,qdr:d" frameborder="0">
</frameset>

search.html



<html>
  <script>
var forum, title, query;
var url = "http://www.google.com/search?q=site:boards.straightdope.com";
var tbs = "&tbs=frm:1,sts:1";

function UpdateQuery() query = document.getElementsByName("query")[0].value;
function UpdateTitle() title = document.getElementsByName("title")[0].value;

function SearchForum() {
  forum = document.getElementById("forum");
  forum = forum.options[forum.selectedIndex].value;
}

function SearchRecency() {
  recency = document.getElementById("recency");
  recency = recency.options[recency.selectedIndex].value; 
}

function SearchSort() {
  sorting = document.getElementById("sorting");
  sorting = sorting.options[sorting.selectedIndex].value;
}

function DopeSearch() {
  var username = document.search.username.value;
  var from = document.search.from.value;
  var to = document.search.to.value;
  if (username) url += ' %2B"Find all posts by ' + username + '"';
  if (forum) url += ' %2B"' + forum + '"';
  if (query) url += ' ' + query;
  if (title) url += ' intitle:' + title;
  if (recency) tbs += ',' + recency;
  if (from && to) tbs += ',cdr:1,cd_min:' + from + ',cd_max:' + to;
  if (sorting) tbs += ',' + sorting;
  parent.google.location = url + tbs;
}
  </script>
  <center>
    <h1>Dope Search</h1>
    <form onsubmit="DopeSearch()" name="search">
      <select id="forum" onchange="SearchForum()">
        <option value="All forums" selected="selected">All forums</option>
        <option value="About This Message Board">ATMB</option>
        <option value="Comments on Cecils Columns">CCC</option>
        <option value="General Questions">GQ</option>
        <option value="Great Debates">GD</option>
        <option value="Cafe Society">CS</option>
        <option value="The Game Room">TGR</option>
        <option value="In My Humble Opinion">IMHO</option>
        <option value="Mundane Pointless Stuff I Must Share">MPSIMS</option>
        <option value="The BBQ Pit">The Pit</option>
      </select>
      User: <input type="text" name="username" size="13%">
      <select id="recency" onchange="SearchRecency()">
        <option value="All times" selected="selected">All time</option>
        <option value="qdr:y">Last year</option>
        <option value="qdr:m">Last month</option>
        <option value="qdr:w">Last week</option>
        <option value="qdr:d">Last day</option>
      </select>
      From: <input type="text" name="from" size="7%">
      To: <input type="text" name="to" size="7%"> (M/D/YYYY) <br/><br/>
      Title: <input type="text" size="20%" name="title" onchange="UpdateTitle()">
      Query: <input type="text" size="30%" name="query" onchange="UpdateQuery()">
      <select id="sorting" onchange="SearchSort()">
        <option value="sbd:0" selected="selected">Sort by relevance</option>
        <option value="sbd:1">Sort by date</option>
      </select>
      <input type="submit" value="Search">
    </form>
  </center>
</html>

Just save the files to your local hard disk and then enter the path to index.html on your local disk into your browser.

I’ve had weird results with Google date range search before in other contexts. In fact I’ve always found it totally unreliable, the results it returns often veer all over the time range and don’t reflect the parameters at all.

This new version:

[ul]
[li]Returns 100 results per page[/li][li]Guarantees the result occurs in the forum you specify (unfortunately demonstrating how spotty google’s index is so far, but that’s no fault of googles IMO - please submit a sitemap!)[/li][/ul]
index.html
unchanged

search.html



<html>
  <script>
var forum, title, query;
var url = "http://www.google.com/search?q=site:boards.straightdope.com";
var tbs = "&num=100&filter=0&tbs=frm:1,sts:1";

function UpdateQuery() query = document.getElementsByName("query")[0].value;
function UpdateTitle() title = document.getElementsByName("title")[0].value;

function SearchForum() {
  forum = document.getElementById("forum");
  forum = forum.options[forum.selectedIndex].value;
}

function SearchRecency() {
  recency = document.getElementById("recency");
  recency = recency.options[recency.selectedIndex].value; 
}

function SearchSort() {
  sorting = document.getElementById("sorting");
  sorting = sorting.options[sorting.selectedIndex].value;
}

function DopeSearch() {
  var username = document.search.username.value;
  var from = document.search.from.value;
  var to = document.search.to.value;
  if (query) url += ' ' + query;
  if (title) url += ' intitle:' + title;
  if (username) url += ' %2B"Find all posts by ' + username + '"';
  if (forum) url += ' %2B"Board > Main > ' + forum + '"';
  if (recency) tbs += ',' + recency;
  if (from && to) tbs += ',cdr:1,cd_min:' + from + ',cd_max:' + to;
  if (sorting) tbs += ',' + sorting;
  parent.google.location = url + tbs;
  return true;
}
  </script>
  <center>
    <h1>Dope Search</h1>
    <form onsubmit="DopeSearch()" name="search">
      <select id="forum" onchange="SearchForum()">
        <option value="All forums" selected="selected">All forums</option>
        <option value="About This Message Board">ATMB</option>
        <option value="Comments on Cecil's Columns">CCC</option>
        <option value="General Questions">GQ</option>
        <option value="Great Debates">GD</option>
        <option value="Cafe Society">CS</option>
        <option value="The Game Room">TGR</option>
        <option value="In My Humble Opinion">IMHO</option>
        <option value="Mundane Pointless Stuff I Must Share">MPSIMS</option>
        <option value="The BBQ Pit">The Pit</option>
      </select>
      User: <input type="text" name="username" size="13%">
      <select id="recency" onchange="SearchRecency()">
        <option value="All times" selected="selected">All time</option>
        <option value="qdr:y">Last year</option>
        <option value="qdr:m">Last month</option>
        <option value="qdr:w">Last week</option>
        <option value="qdr:d">Last day</option>
      </select>
      From: <input type="text" name="from" size="7%">
      To: <input type="text" name="to" size="7%"> (M/D/YYYY) <br/><br/>
      Title: <input type="text" size="20%" name="title" onchange="UpdateTitle()">
      Query: <input type="text" size="30%" name="query" onchange="UpdateQuery()">
      <select id="sorting" onchange="SearchSort()">
        <option value="sbd:0" selected="selected">Sort by relevance</option>
        <option value="sbd:1">Sort by date</option>
      </select>
      <input type="submit" value="Search">
    </form>
  </center>
</html>


I got a warning a while back for suggesting specific technical solutions to some SDMB problems. Tuba cited an unwritten “junior admin” rule. Just sayin’ … I don’t think they’ve ever implemented any specific solutions from users in the past.

Silly rules aside, this is made for you to save to your own computer and use from there.

I copied and pasted the files, and they worked amazingly well. If Google would index the entire SDMB, this would be an incredibly useful tool.

I sent up the newest versions. I got these error messages from IE8:

Webpage error details

User Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Timestamp: Fri, 7 Aug 2009 21:25:09 UTC
Message: Expected ‘{’
Line: 7
Char: 24
Code: 0
URI:
Message: Access is denied.

Line: 52
Char: 54
Code: 0
URI: http://www.google.com/extern_js/f/CgJlbhICdXMrMAo4J0AILCswDjgFLCswFjgOLCswFzgDLCswGDgELCswGTgGLCswJTjJiAEsKzAmOAUsKzAnOAIs/2NzrGnsQToA.js

(I had to delete the first URL because it is a proprietary site.)

Does your code have to be in the root folder?

They can be in any folder, doesn’t matter which one as long as both files are in the same one. I only test in Firefox and Safari. It looks like IE8 is doing some kind of hyper-anal access control restriction where you can’t embed any third party site in a frame. It might also be choking on my inline function definitions. Maybe adding { } to lines 6/7 would fix that one.

Sorry to be such a pain in the ass, alterego. :frowning:

Looks cool, alterego! Thanks!