How to stop Google from crawling secure content/directory?

[Ozzman]

My post title is clearly stating about my question . But lets repeat it. How can we stop google bots to crawl through a secure (not-willing-to-share) information from one of the website's directories.

Can we use robots.txt for this or you may suggest any other better treatment for this?

If Google crawls any secure information from a website content and shows it in seacrh engines then what can we do to let this information disappeared from Google search results?

Your co-operation is appreciated in adavance.

[thindenim]

Add the following to your robots.txt file, where directory is the area you want to disallow. This tells the robot they can't access or index this section.

User-agent: *
Disallow: /directory/*

Alternatively you can use the meta noindex tag on each page: -

<meta name="robots" content="noindex, nofollow">

which will mean that the page is not indexed and links not followed, or: -

<meta name="robots" content="noindex, follow">

which will mean that the page is not indexed, but any links are followed

[fernimac]

The previous post has been very clear. Although I would add that if your pages are in a secure area, Google should not be able to get to those pages. Even if you include a noindex no follow directive, users could still get to those pages. You should implement a secure access via a password instead so that no SE and no undesidered users get to your private pages.

[thindenim]

Good point fernimac, you should always password protect sensitive information.

[Palindrome]

Hi

All good advice has gone before me.

In case you are taking the robots.txt route, best to use the meta tag as well. If a link somehow exists to a page, or is created by accident, that could still be crawled regardless of robots.txt.

[Peter (IMC)]

This is one of those phylosofical questions. Googlebot is nothing more than a normal visitor who, when asked, will refer others to those pages.

If you allow visitors to that part of the site without the need to login, how are you going to prevent them from refering their friends to that part of the site?

If you really want "secure (not-willing-to-share) information" to be available to only those that you choose, you need to password protect it.

robots.txt or other "just for the search engines" kind of ways aren't the way to go because they aren't meant to block "not allowed" visitors.

[Jean-Luc]

robots.txt and meta tags are not appropriate ways to protect confidential information. Confidential pages should be password protected.

robots.txt and meta tags are only meant to pass information to well-intended robots and search engines. Some ill-intended bots will use it to detect potential weaknesses in your web site.

If your private pages are already in Google, visit How can I prevent my own content from being indexed or remove content from Google's index? (see the part about expedite removal).

Jean-Luc

[adverlicious]

Google provides definitive instructions for: (1) removing sensitive pages from Google's search results, and (2) preventing pages from being indexed by them in the first place:

Preventing content from appearing in Google search results

Note that, with the exception of Google, Yahoo!, MSN, and Ask, you should expect both your meta tags and requests for page removals to essentially be ignored. Worse, many of these "rogues" are based overseas where you'll have little or no legal recourse in the event of a problem.

Relatedly, don't assume that your private pages are "safe" from being Google-indexed because you have no links pointing to them or you haven't submitted them. Search engines have many, many ways of finding new pages -- e.g. a visitor's Google toolbar auto-submitting them, a competitor submitting them for you, links from sites you don't control, etc.

Confidential info should always, always be password protected -- or never placed online at all.

[edhan]

Yes. The best solution will be password protection for that directory. Using nofollow or noindex may stop the crawlers but humans still can access them.

[incrediblehelp]

Sure using password protection is best, but you still may be able to link to certain pages with some password protection scripts. You can also disallow viewing directories through the htaccess file on Apache web servers

[johnehogan]

As a systems admin, I agree that using a simple .htaccess file with a corresponding .htpasswd file is the ONLY way to protect a sensitive Directory (or folder) against prying eyes. This is easy to do on Linux boxes. The folder CAN still be accessed by those who KNOW what the password is for the folder, but search engines and others will be locked out totally

[kurt.santo]

johnehogan,
Great input!

I worked with .htaccess, but not with .htpasswd. Would you have an entry in .htaccess naming the directory that is protected and then have in .htpasswd the password and usernames stored? And if yes, how do you do this in detail?

Thank you,
Kurt

[Fibo]

Everything that you put on the web might be found by a human or robot visitor... So do NOT place any "really need to really be secure" information on the web.
The robots.txt is NOT the good solution: it just says "this is a secret area, PLEASE don't come". Well educated spiders will respect your secret, bandit spiders too -but by exploring first this advertised secret area!
So a bare minimum would be the robots.txt exclusion AND an index.htm file in the directory, that silently redirect to your homepage (no sound, no noise, do not alert the bandits), maybe with a 301 redirect.
Better is the HT protect with .htaccess and its password file (usually named .htpassword, but this name is not fixed).
Best is: don't put it on the web

[Webnauts]

I think I have a solution for you: Preventing Search Engine Indexing of Secure Pages - SEO Workers

[g3 creative]

To stop Google from crawling secure content you should always use password protection as standard.

Dave Mac

G3 Creative

[Webnauts]

Originally Posted by g3 creative

To stop Google from crawling secure content you should always use password protection as standard.

Dave Mac

G3 Creative

And what is if you do not want the pages to be password protected, but still don't want those pages to be crawled?
I provided a solution above (my tutorial), but seems it have been ignored.

[Peter (IMC)]

but seems it have been ignored.

oh stop crying every time you don't get a standing ovation. Your posts are read and you just have to get used to the fact that most people don't fall on their knees to thank you.

You should watch the movie "the secret". Even though I think it's a dumb ass movie, they are right that if all you want to see is negative, all you will see is negative.