View Full Version : Robots
santos007
06-10-2004, 04:33 AM
Our Company, among other activities, we offer SEO services, we have around 30 first in Google, of differente clients and more than 60 in the first page of results. However, we are at a loss regarding how to deal with robots. could someone recommend me a program to deal with robots? 1) Knowing robots, 2) Disallowing some of them, 3) Closing some pages to Robots.
Thank you very much.
nikao
06-10-2004, 06:10 AM
Our Company, among other activities, we offer SEO services, we have around 30 first in Google, of differente clients and more than 60 in the first page of results. However, we are at a loss regarding how to deal with robots. could someone recommend me a program to deal with robots? 1) Knowing robots, 2) Disallowing some of them, 3) Closing some pages to Robots.
Thank you very much.
uhm.. just a regular robots.txt ?
http://www.robotstxt.org/wc/norobots.html
DanMagicMarvel
06-10-2004, 09:20 AM
I have found Robots Manager to be an absolute must have for myself. http://www.websitemanagementtools.com/robot-manager/professional.php
Not only will it help you in creating Robots.txt (helpful for really large sites) - But it can analyze your log files and give you a report on spider activity. Which ones visited your site, which pages, which pages were indexed, etc...
It's a gem.