Re: Can The Spiders Eat?
Yes, spiders can access password protected content. Basically, you would need to set the site up so that it recognizes traffic from the spiders, and serves them the content, only presenting human users with the username/password prompt. You would then add a no-cache meta tag so that the page does not show up in Google's public cache. I see this a lot, especially with forums that require registration to read the posts, but that want the traffic from Google.
I report them to Google as spam.
__________________
The best way to learn anything, is to question everything.
|