As far as I'm aware, unless you stipulate in your robots.txt that a folder/files should be disallowed, the S/E robots will crawl the whole site.