Would I be blocking Google?

[business-directory]

Does anyone know whether we would be locking Google out from indexing some of pages if we required a cookie? In other words; if no cookie existed, we would block access. Does Google accept a cookie when they spider, and use that cookie throughout the session?

The reason for wanting to do this is to keep some email harvesters out of our site Australian Search Engine and Business Directory, find businesses in Australia clickfind™

[activeco]

Wherever possible, avoid the use of session IDs in URLs. Consider using cookies instead.

However: What are URLs not followed errors?

[wige]

Google and search engine spiders in general are stateless. They do not accept, parse or process cookies. Or to apply a different line of thinking, you are requiring cookies to restrict access to spiders, which are built in essence to work the same way that search engine spiders do.

[business-directory]

Activeco; thanks for your reply, but we are not using sessions in the URL, we are using cookies.

Wige; cheers - I have to ask, do you have proof to back up your statement that spiders are stateless? I want to believe you, but it will mean that I'll have to come up with something else to stop email harvesters from access (

[business-directory]

Activeco; Aah, I just followed the link you provided, and I see that Google answered the question. Thats a shame. Back to the drawing board... Cheers guys

[wige]

Originally Posted by business-directory

Wige; cheers - I have to ask, do you have proof to back up your statement that spiders are stateless? I want to believe you, but it will mean that I'll have to come up with something else to stop email harvesters from access (

The only way to prove it would be to set up your logs to show the cookies contained in requests, and you will note that the spiders never send a request containing a cookie, which is the mechanism that maintains state. (The web is really stateless, the only way to simulate state other than AJAX is with cookies) In addition, if you use sessions, a session will typically expire before the next visit of a spider.

[activeco]

Activeco; Aah, I just followed the link you provided, and I see that Google answered the question.

Yes, it pays off reading to the end.

BTW, the first sentence comes from Google too, which implies that they mostly have no (indexing) problems with cookies.
However as Wige points out, you don't see cookies in requests, which means you can't provide further content based on the cookie.

The problem with session id is that it creates different url for every new visit and that could endanger proper indexing of a site.

[bunty]

The problem with session id is that it creates different url for every new visit and that could endanger proper indexing of a site.