View Single Post
  #1 (permalink)  
Old 07-01-2007, 01:10 PM
jb1702 jb1702 is offline
WebProWorld New Member
 
Join Date: Jul 2007
Posts: 2
jb1702 RepRank 0
Default robots.txt unreachable - googlebot stopped crawling

Hallo everyone
My website has been reguralry crawled by googlebot
for two years now. Suddenly on June 8 everything stopped because of a
missing robots.txt file. Until then googlebot was crawling the site
daily and I had over 10.000 pages in the index. I did't have a
robots.txt and my server returned 404s

GET /robots.txt HTTP/1.1" 404 2157 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1...

On that day googlebot asked for the robots file, crawled some pages
and never returned (I never got another googlebot request according to
my logs).

On my webmasters tools now I have:
robots.txt URL http://www.tospitimou.gr/robots.txt
Last downloaded June 8, 2007 2:04:45 AM PDT
Status 404 (Not found) [?]

and almost 5.000 "Unreachable URLs" errors, with cause "robots.txt
unreachable". I have added a robots file about two weeks now and
changed it a few times in that period too, but googlebot has never
made another request for anything since that day. Yet many of the
"Unreachable URL" erros where registered during that time, but not a single
page or robots request in my logs.

Can you think of anything that could get me out of this?

Thanks in advance

Last edited by jb1702; 07-01-2007 at 01:12 PM.
Reply With Quote