View Single Post
  #9 (permalink)  
Old 02-21-2008, 08:04 AM
Conficio Conficio is offline
WebProWorld Veteran
 
Join Date: Jul 2003
Location: Mass, U.S.A.
Posts: 399
Conficio RepRank 0
Default Re: Google ignores robot block

DON'T PANIC!

I think you have to better understand how a web crawler like Googlebot works. It gets a huge list of pages to crawl and follows it religiously. When you put a meta tag on your page (or pages) then Google does not stop crawling immediately if it already has the URL's in its list. Later it will analyze the pages read in the crawl and eliminate the URLs from its crawl list, as to your instructions, and also see which pages it should add to the index.

Remember, Google needed to read you page in order to even discover your instructions in the meta-tag.

All what happened so far is that the Google Bot did crawl a few pages. That in itself is not evidence that it will include it in the index (prevented by "NOINDEX") or does follow the links from this page for further crawls (prevented by "NOFOLLOW"). also it takes weeks for Google to purge pages from its index tagged by NOINDEX.

Actually, Google will come back to crawl pages that have links to it from the outside world. And there are some automatic links from sites like About US, which makes a wiki page for every domain registered.

To accelerate the purging of your pages from the Index, sign up for a Google Webmaster account and use Tools --> Remove URLs. Unfortunately ,for that to work you have to first authenticate your site.

Good luck

K<o>
Reply With Quote