Blocking Search Engines from web pages

I have a small website where I list parts and services that I offer. It is designed to show what my products and services are to my wholesale customers, but is also safe for retail customers to look at because I have no pricing listed.

I added one page that has wholesale pricing listed so my wholesale customers can check their costs out, but I made the link unmarked and hidden. I also set up a small html code so that if someone clicks on the link, they get a page that requires a password to get to the pricing page. It is not a secure system for sure, but it does not concern me that much because non-wholesale customers can’t get the pricing without being a dealer anyway.

The problem is, if you Google my domain name, one of the page views that is available is the wholesale pricing page. I would rather this information not be that easily found.

Is there a way to add some code or something to a web page so that the search engines ignore the page or bypass it?

Well, if you don’t want the wholesale price data to be publicly accessible, you should actually make it so that it is not publicly accessible. It sounds like your password prompt relies on client-side JavaScript, which is basically like handing the user an envelope containing your password, and saying “What’s the password? No peeking in the envelope!” :stuck_out_tongue:

To make information only accessible to authorized users, you need to configure your security on the server side of things. How exactly you go about doing this will depend on what type of server is hosting your website, how much access you have to it, and exactly what system of security and authentication you want to implement.

If you’re running Apache, one really simple way to password-protect a page is to use .htaccess and .htpasswd files to only allow users to retrieve pages from a certain directory if they are authenticated with a password. This is easy to do, and works well as long as you don’t need a “real” user management system. I won’t go into details since I’m not sure what sort of server you have, but if you are running Apache and can get access to a command line I could walk you through the steps.

To answer your original question, though: to prevent search engines from indexing your pages, you can use a robots.txt file to tell them which pages they are and are not allowed to index. This will work with all the major search engines, but do note that this isn’t a security measure - it does nothing to hide your data; it’s more of a polite request to stay out of certain areas.

There is also an easier way, if you have only the one page you don’t want indexed.

Insert this in the head section of that page:

Again, this is only a request to the Search Engine robots to ignore this page. But well-behaved robots, like those from Google, Yahoo, etc. will follow such requests.

Thanks for the helpful information .

I know the page isn’t secure, and it doesn’t really have to be. Since most people will not know the page exists, and it isn’t obvious where the link is, I don’t think the casual person will find their way to the one hidden page. It won’t stand up to a real search, but the info is not critical. It is just that Googling my domain name and coming right up with a link to the hidden page is just too simple. At least is the search engines don’t look at it and list it out like that, there is far less chance that anyone will find it.

I would never put any really important info on a page so flimsily protected. Just trying to make it less obvious that the info is there except by the people I clue in to it.

Now that a simple Google search can find the page, it’s confidentiality is gone. You can attempt to remedy that by removing the page in question, or move it to a more secure area of your site as mentioned earlier in the thread so that subsequent Google bots cannot find it and its priority drops in ranking. What is important is that the longer it remains where it is, the less likelihood you will ever be able to hide it.

While you are not really concerned that a casual snoop might find it, what about your business competitors? Perhaps that may not matter to you. Then again, what if a competitor finds it and decides to undercut all of your prices? In this economic climate keeping just one customer from straying might be important, even if you are only undercut by one cent. Also, an inadvertent security lapse that remains (especially when the owner knows and does nothing to correct it) goes to the credibility of the site owner, too. If I was a customer and found out you had a simple security lapse on your web site, and you knew and took no steps to correct it, I might wonder how you address confidentiality in other areas of your business. That alone might drive me away to a competitor, even if I have to pay more, because my business privacy is more important to me than the dollar price I pay for one of your widgets.