No spider should ever find the index.htm file in a subfolder unless there is a link going to that folder, perhaps a broken link like
http://www.mydomain.com/images/
Although the robots.txt file is useful, it is not reliable. I've seen Google and Yahoo index pages that were excluded properly in the robots.txt.
The method that I use that thus far has been 100% reliable is to use the robots meta tag with a noindex attribute. Whenever I use an index.htm file to protect a directory, it includes the following:
<meta name="robots" content="noindex,follow">
I also include a link back to the home page.