Quote:
|
Originally Posted by incrediblehelp
Not sure if the spiders will follow a robots.txt in a subfolder. Usually the spider follows the first robots.txt it sees in the root directory. Can you post it to review.
|
Spiders obey to robots.txt in subfolders, but only for the content (files) of the subfolders.