Google would not penalize a site for getting too many inbound links at once. If they did, then any of us could ruin our competitors by adding their websites to hundreds of spammy directories and link farms. Google realizes that you cannot control who links to you, so how could they penalize you for something you have no control over?

If you lost a lot of good inbound links, that is another story. Google does still weigh the number of links that are pointing to your website. But if I added a link to your website to 10,000 spammy directory pages tomorrow just because I didn't like your site - would Google or any other search engine penalize your ranking for that? It just doesn't make any sense that I (or anyone else) could have that much power over your rankings.

On the other hand, you CAN control who you link to. You can also control the content of your site. These would be the things that I would look into if I were you. Things happen. If 10 of your competitors add new content to their sites and you don't, your site could easily slip in the search engine rankings.