Is there a way to automate searching a website for multiple items?

General Questions
#1

The idea comes from me having a lengthy list of movie titles and I’d like to search for them as a batch on a site like instantwatcher.com or netflix, or IMDB. It seems to me, perhaps incorrectly, that there might be a way to create a script or bookmarklet or something to automate the search and collect data on which searches got a hit?

It would be kinda cool to find a list on say “20 things you must have when camping” and then search Amazon for each item all at once, maybe with results opening in tabs or something.

Am I just imagining that this sort of thing can be done? It seems like something that someone with the right know how could make work. I don’t really know where I’d start searching for how to do it either.

#2

I 'm getting a feeling that the answer is no…

#3

I can see something like this maybe arising at some future point out of efforts to get computers to understand things semantically and contextually; i.e. “Grandchild of Siri; find me some good prices for all these things”

But for now, unless the blogger who wrote/plagiarised the list included search links, you’re probably going to have to do some legwork still.

On further thought…
Maybe you could embed the page in an iframe, surrounded by a stack of context-sensitive ad banners. Not sure if that would actually work…

#4

Of course there’s a way. With computers there’s always a way. The questions you should be asking though are how can I do this and how much time and effort will it take me?

#5

Netflix and Amazon have APIs. You can sign up for an API key and write a quick-and-dirty piece of software in your favorite language to do this, no problem.

So yes it can be done. How easy it is to accomplish depends on how experienced you are at writing programs that make use of web APIs.

Edit: Just checked, IMDB has an API also.

#6

I may have misunderstood what you meant by ‘list’. Are you talking about a neat, finite list of ready-made search terms, or a blog post ‘list’ where the keywords are buried in pages of waffle?

#7

You don’t even have to use APIs, you can write code that sends HTML page requests that look exactly the same as what your browser sends when it does searches on those sites. Well, when I say “you” I mean “a programmer,” which will be the issue for most people.

#8

Page scraping like that is frowned-upon for two reasons:

  1. You’re not gathering data with the permission of the database owner, which is mildly unethical
  2. Your code is a lot more fragile, and even a tiny change to Amazon’s code could cause it to break

If there exists an API, use the API. If not… well, use your judgement. But in this case, they all have APIs.

#9

This would be a super clean list like a text file with values separated by returns, or a CSV, or anything else Excel can crank out.

#10

Valid discussion points but I was just addressing what is possible technically.

I used to work for Network Solutions and there were businesses out there whose entire business plan hinged on doing this to our web site. We didn’t mind, but when we redesigned everything they screamed bloody murder and our response was basically, “Sorry you took that risk.”

#11

You can do such things without APIs, using a GUI automated test tool.