robots.txt unreachable - googlebot stopped crawling

[jb1702]

Hallo everyone
My website has been reguralry crawled by googlebot
for two years now. Suddenly on June 8 everything stopped because of a
missing robots.txt file. Until then googlebot was crawling the site
daily and I had over 10.000 pages in the index. I did't have a
robots.txt and my server returned 404s

GET /robots.txt HTTP/1.1" 404 2157 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1...

On that day googlebot asked for the robots file, crawled some pages
and never returned (I never got another googlebot request according to
my logs).

On my webmasters tools now I have:
robots.txt URL http://www.tospitimou.gr/robots.txt
Last downloaded June 8, 2007 2:04:45 AM PDT
Status 404 (Not found) [?]

and almost 5.000 "Unreachable URLs" errors, with cause "robots.txt
unreachable". I have added a robots file about two weeks now and
changed it a few times in that period too, but googlebot has never
made another request for anything since that day. Yet many of the
"Unreachable URL" erros where registered during that time, but not a single
page or robots request in my logs.

Can you think of anything that could get me out of this?

Thanks in advance

[Webnauts]

Your robots.txt as they are now cannot be a problem. From what you claimed above, or you have broken links or your server was not available.

Another post possible problem can be your DNS errors: DNS Stuff: DNS tools, DNS hosting tests, WHOIS, traceroute, ping, and other network and domain name tools.
You better check that out with your provider as soon as possible.

When all issues above are clarified and solved, then create an XML site map and submit it to Google.

I just checked and you are not banned from Google.

Good luck patrida! I am half Greek. LOL

[jb1702]

Euxaristw for your answer

Well, I know about my dns issues, they were there from the beginning and they never caused any problem. I also used to have a sitemap, but I removed it after the problem with google appeared. Now, whenever I try to upload a new one, google says it's unreachable too.

What puzzles me is that I have absolutely no Googlebot requests in my access and error logs. I also tried to translate some pages of my site using the google language tools and they were unreachable from there as well.

So what I'm thinking is that somehow the google ips have been blocked from by server. Do you know of a way to verify what ips are blocked from my host's machine?

Originally Posted by Webnauts

Your robots.txt as they are now cannot be a problem. From what you claimed above, or you have broken links or your server was not available.

Another post possible problem can be your DNS errors: DNS Stuff: DNS tools, DNS hosting tests, WHOIS, traceroute, ping, and other network and domain name tools.
You better check that out with your provider as soon as possible.

When all issues above are clarified and solved, then create an XML site map and submit it to Google.

I just checked and you are not banned from Google.

Good luck patrida! I am half Greek. LOL

[Webnauts]

I see you are on an Apache server. You should check if the IPs are blocked in your .htaccess file.

[Webnauts]

I cannot imagine that your site is crawlable anyway, if you have on the homepage alone 1490 HTML errors: Result for http://www.tospitimou.gr/main/index.jsp - W3C Markup Validator

For God's sake man.

[Webnauts]

And this is damn weird brother: Tospitimou.gr - To Spiti Mou

I get: %ERROR:101: no entries found

You seriously need professional help!