Quote:
|
Originally Posted by computergenius
Hi Seminole386
I notice that your robots.txt is missing, and when I ask for it I get redirected to your home page - presumably because all missing pages get redirected to your home page.
I have often wondered if this treatment of robots.txt is harmful, as I assume that the robots get confused...
I wonder if anyone knows?
|
Robots.txt must be there, and for the purposes intended. Google has good writeup on how to help their spyders with robots.txt entries.
Redirects for 404 errors should go to an error page, with links to the main sections of your site, to help visitors get back on track.
You should pick one or the other in regards to your two URLs that point to the same place.
(All my opinion)
(hope this helps)
-John