what is GOOGLE BOT?

[namita]

What is googlebot meta tag? what does it stand for/ what does it do?

<META NAME="Googlebot" CONTENT="index,follow">

Are we trying to tell the google spiders to index the pages?

[webmastery]

Yeah, that's what it is, but it is useless. GoogleBot already knows to follow so there is no need including that tag.

[nakulgoyal]

GoogleBOT is Google's Spider Identification Name that our servers detect as the Search Engine's Spider.

There are a lot more. Check from the robots.txt file here at www.seomasters.com/robots.txt

--
Nakul Goyal
MCP, Brainbench 'MVP', BCIP, CIW Associate
SEO / SEM / SEP & Link Building Consultant
www.nakulgoyal.com
www.cwsteam.com
www.seomasters.com

[LoveNotWar]

Please can someone explain to me why one would not want some bots to spider ones site?

I would have thought that the more spidering that happens on your site the better, no? But I must be missing somthing because it seems quite common to disallow certain ones

[minstrel]

Please can someone explain to me why one would not want some bots to spider ones site? I would have thought that the more spidering that happens on your site the better, no? But I must be missing somthing because it seems quite common to disallow certain ones

See http://www.robotstxt.org/wc/active/html/type.html.

1. to hide swiped content from copyright checkers?
(e.g., "unnamed - Purpose: Copyright Infringement Tracking")

2. because you believe that some come with bad intent, like email harevesters (e.g., "e-collector - Purpose: email collector"; "DownLoad Express - Purpose: graphic download")

3. because some exist for the purpose of facilitating someone downloading your entire site for "offline viewing" and maybe that worries you (e.g., Webreaper)

4. because your site or one or more subdirectpries on your site isn't intended for public viewing, for any number of reasons (e.g., family information and photos to be shared with other members of your family only)

Mind you, I don't know that there's any guarantee that the robot you try to disallow will actually pay attention to that instruction, especially if you're right about bad intent.

[nakulgoyal]

I think David best clears it. The point at the bottom is IMPORTANT.

Rest robotsorg is there to help as a resource. You cal always post in here to know more user/expert comments

Nakul Goyal
MCP, Brainbench 'MVP', BCIP, CIW Associate
SEO / SEM / SEP & Link Building Consultant
www.nakulgoyal.com
www.cwsteam.com
www.seomasters.com