Scary Warnings in Webmaster Tools?

[keyon]

Whenever I create a password-protected directory on my site, Webmaster tools gives me kind of a scary-looking "warning" that they couldn't access the pages. Is this just a heads-up for my own benefit, or is Google telling me I did something wrong?

[SteveGerencser]

It's just them complaining that they can't get to pages they want to crawl. Google wants your data.

[keyon]

It's just them complaining that they can't get to pages they want to crawl. Google wants your data.

Thanks, Steve. No more wondering about that. I would think it's pretty common to have protected directories in a website.

[weegillis]

Thanks, Steve. No more wondering about that. I would think it's pretty common to have protected directories in a website.

Not just directories, but documents, some of which are found in public directories, such as short lived PDF, etc.

For these types of files, it is common to see a cgi redirect that robots can't follow. WMT has always complained about this. How else can a person keep them out of the index?

[keyon]

WMT has always complained about this.

Guess I don't care if Google complains about it, as long as they're not doing something weird like knocking my position in the serps because of it.

[weegillis]

It might be they would prefer we use robots.txt to restrict individual documents. Good for them, but more work for us to maintain as each document crops up and later expires. The cgi redirect is what I settled upon as it matters not the robot, they none of them can traverse the cgi-bin folder. It also gives me chance to log it along with the other page views.

[kgun]

Guess I don't care if Google complains about it, as long as they're not doing something weird like knocking my position in the serps because of it.

You can allow Bots access on a password protected site. I know the phpBB platform very well. On that platform there is an option called bot access. It should not be so difficult to copy that code if you want one or more bots to index your password protected areas of your site. But that is perhaps not what you wan't.

Be aware though:

Is your online password protected database on a secure server really secure?

[MyRentals]

Informative comments are also listed in this blog article about Allowing Google Into Password Protected Areas

The comment I particularly thought was relevant to the above post IF you have Adsense blocks within the protected area was by a reader named Mark:

If you wish to give Google access to the restricted areas of your site for adsense purposes Google has the tools.

Go to Adsense, Adsense Setup, then Site Authentication.

From here you can give the Google bot POST username/password text that it can use to log in and index for the purpose of serving ads. This doesn’t help for getting content into the main index but is nice if you want relevent ads for your subscription material.

Like David has said. If it is indexed it can be seen. So this is why google doesn’t offer this service for regular indexing.