Quote:
|
Originally Posted by calebtheredwood
thanx for your replies!
could i just add a robots.txt to dis-allow them to crawl any of the freewebs version? or would i risk being removed from yahoo all together?
|
Not sure if the spiders will follow a robots.txt in a subfolder. Usually the spider follows the first robots.txt it sees in the root directory. Can you post it to review.