Hallo everyone
My website has been reguralry crawled by googlebot
for two years now. Suddenly on June 8 everything stopped because of a
missing
robots.txt file. Until then googlebot was crawling the site
daily and I had over 10.000 pages in the index. I did't have a
robots.txt and my server returned 404s
GET /
robots.txt HTTP/1.1" 404 2157 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1...
On that day googlebot asked for the robots file, crawled some pages
and never returned (I never got another googlebot request according to
my logs).
On my webmasters tools now I have:
robots.txt URL
http://www.tospitimou.gr/robots.txt
Last downloaded June 8, 2007 2:04:45 AM PDT
Status 404 (Not found) [?]
and almost 5.000 "
Unreachable URLs" errors, with cause "
robots.txt
unreachable". I have added a robots file about two weeks now and
changed it a few times in that period too, but googlebot has never
made another request for anything since that day. Yet many of the
"
Unreachable URL" erros where registered during that time, but not a single
page or robots request in my logs.
Can you think of anything that could get me out of this?
Thanks in advance