Excluding the "good" BOTS from /images folder
Hi again all! I had an issue way back with MSN trying to index my site from my /images/ folder due to the presence of an index.htm file
Since then I have excluded the MSN bot from the /images/ and VIOLA! my site is now being indexed properly!
Today I discovered that Google has my /images/index.htm page in a result for a keyword of mine. Mind you it's a pretty obscure keyword but nonetheless, it's been indexed. I see no evidence of Google trying to find my content within the /images/ folder as the MSN bot kept trying.
Question: Should I exclude the /images/ directory from all bots? If I do that then won't the cached pages not show the images? I tried to disable hotlinking once and the cached pages looked awful! Would you exclude the /images/ folder or is there another way that I can stop the contents of said folder from being displayed when some snarky user wants to see the images in the directory?
Many thanks!
Maggie
|