search engine spider behavior

[Open_Your_Eyes]

Hello. To properly display page elements, I often have to check the user's browser type. What would a search engine spider return for such a test? Would it just be unrecognizable (Unknown/Various 0.0) as no actual web browser was being used?

[bhartzer]

That's a good question. I had to check my logs myself to remember exactly how their user agent is listed.

For Google, you'll see this as a typical log file entry:

2004-11-29 01:51:37 66.249.00.00 - www.yourdomain.com 80 GET /index.html - 200 Googlebot/2.1+(+http://www.google.com/bot.html)

Google's user agent is shows as their bot address (www.google.com/bot.html) so you can get more information about them. The user agent is what's in the parenthesis.

If someone visits using a web browser such as IE, it will look more like this:

2004-11-29 01:52:40 68.95.0.0 - www.yourdomain.com 80 GET /index.html - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1)

The user agent in this case is Mozilla compatible, or MS Internet Explorer 6.0.

You might want to look at your log files in particular to see what they're going to show because each web server is configured differently, especially when it comes to the data that's being saved.

In your case, I would suspect you'll want to recognize certain web browsers (Netscape, Firefox, and IE) and then have give another page for ones that you don't recognize, which would include the search engine bots.