Nope. Not at all. Sitemaps should have no direct effect on rankings - they affect crawls by telling the search engines when content is updated, as well as providing the search engines with deep information about your site. However, your site should not be so complex search engines have to use the site map to find your pages.
Google, MSN and Yahoo have all stated they support Sitemaps. Not sure about Ask.

__________________
The best way to learn anything, is to question everything.