View Single Post
  #7 (permalink)  
Old 03-24-2008, 10:17 AM
wige's Avatar
wige wige is offline
Moderator
WebProWorld Moderator
 
Join Date: Jun 2006
Location: United States
Posts: 2,661
wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9
Default Re: Can The Spiders Eat?

Yes, spiders can access password protected content. Basically, you would need to set the site up so that it recognizes traffic from the spiders, and serves them the content, only presenting human users with the username/password prompt. You would then add a no-cache meta tag so that the page does not show up in Google's public cache. I see this a lot, especially with forums that require registration to read the posts, but that want the traffic from Google.



I report them to Google as spam.
__________________
The best way to learn anything, is to question everything.
Reply With Quote