View Single Post
  #13 (permalink)  
Old 02-05-2008, 02:32 AM
Dubbya's Avatar
Dubbya Dubbya is offline
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Nov 2006
Location: Steinbach, Manitoba, Canada
Posts: 1,300
Dubbya RepRank 4Dubbya RepRank 4Dubbya RepRank 4Dubbya RepRank 4Dubbya RepRank 4
Default Re: Excluding the "good" BOTS from /images folder

I wouldn't even bother with excluding the index.htm files located in your subdirectories via the robots.txt file.

Just use the following Meta tags in each document and you'll be good to go.

Code:
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
<META HTTP-EQUIV="CACHE-CONTROL" CONTENT="NO-CACHE">
<META HTTP-EQUIV="PRAGMA" CONTENT="NO-CACHE">
<META NAME="ROBOTS" CONTENT="NONE">
<META NAME="ROBOTS" CONTENT="NOARCHIVE">
<META NAME="GOOGLEBOT" CONTENT="NOARCHIVE">
Reply With Quote