New Google user-agent?

[minstrel]

GoogleGuy Says: New Google user-agent?

March 5, 2004

I wanted to give you a heads-up about a potential change in our user-agent name. Currently we use the user-agent

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

but we're considering changing our user-agent to something like

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The primary reason for this is that some web servers assume that unless a user agent is IE, Netscape/Mozilla, or maybe Opera, that your browser won't support JavaScript, frames, etc. As Googlebot gets better over time, it gets closer to a regular user and browser in our ability to handle features like that. This user-agent change would let vanilla webservers assume Googlebot is more like a regular user, while still allowing site owners to recognize that Google is visiting and still providing clear contact info in case of questions or issues. We would still respect "Googlebot" in robots.txt, so the overwhelming majority of webmasters wouldn't have to change anything at all; most site owners probably wouldn't even notice the difference. This bot that a few people noticed was a test crawl with the new user agent. It looks like things went very smoothly, but we'll still be conservative when making this change. If people have any positive or negative comments or questions, just post here. If we don't hear any showstopping objections, we'll look at making this name change on the user-agent when we're convinced everything would be a smooth transition.

thanks,
GoogleGuy