Specifically, products that really only exist as database items until someone calls them in a search?
Amazon, for example, doesn’t actually have static web pages for each of their products; the “pages” are created dynamically based on user inputs.
So how does Google index these products? It seems a bit hit or miss. Sometimes I get Amazon links in search results and sometimes not, even when I know they have the particular item.
The company I work for has a similar site (dynamic content) and they’re asking me if I know how it’s done… I said I would ask the Dopers!
Generally, they spider the site. For instance, lots of people link to the Amazon home page, so Google asks the server for the current home page content regularly to update the search engine.
There’ll be links to some featured products on the home page at any time. So Google will add those “page” addresses to its list, even though they’re dynamic.
And then, when it asks for product X, there’ll be links somewhere to say “Customers who viewed this item also liked products Y, Z, and Q3.” So Google will add those “pages” as well.
That’s why it’s hit or miss; some products Google was able to find without using Amazon’s search box (because it has no idea what to “type in.”) Some items didn’t get spidered.
Spidering is how static pages are indexed too, for what it’s worth. Google doesn’t know that you have a directory called Pages and it has a.html, b.html and c.html in it. It has to go to your site and follow every link it finds.
So it’s all about links… that makes sense. My recommendation to the suits - “Make sure every item can be found by following menus and that you have a Google Site Map.”
And Amazon’s web pages are static. There’s a whole lot of crap appended to the middle and end of every URL, but the base of it–the ASIN and some other stuff–remains the same over time. All of Amazon’s products have the following basic URL format and have for some time: amazon.com/dp/B00006IFKU/