The error in the robots.txt file won't stop spiders from visiting your site, alienzhavelanded - it's just that when they arrive they will see a notice saying "do not index" and leave again, so you would still see "visits" from the spiders in your logs. Unfortunately, all they will be "indexing" is the robots.txt file.
As for why you are still listed in other engines, perhaps that's because they update or "prune" less often... I'm not sure.
|