View Single Post
  #2 (permalink)  
Old 05-09-2004, 04:46 PM
moby302 moby302 is offline
WebProWorld New Member
 
Join Date: Mar 2004
Location: South Africa
Posts: 9
moby302 RepRank 0
Default

I noticed that Alexa has been crawling my site daily, not knowing what crawling was I went to their website and read up on what it was all about, its quite interesting and offers a whole lot of links that show you what you can do about it etc, something to do with making a robot.txt file that speciefies what the robot is allowed to crawl and what its not allowed to do, I believe that you can also stop anyone from crawling with the right .txt file, not quite sure how it works, but I would go to the sites involved and check thru them and see what they say about it, its your right to limit access if you want to.
I just wish all the funny entries on my stats would go away, I am sick of pages of codes that keep on appearing, I cannot delete the whole IP range from the person who is doing it, just block it on my own pc.
Reply With Quote