I had a site with a few broken links and I got penalized. All of a sudden, the site dissapeared from the SERPs and when I went to check out what could have been the reasons, I found those broken links.

It's common sense in my opinion for Google and other search engines that a site with many broken links is a neglected site thus Google will treat it the same way.

Better spend some time... from time to time to check your links and avoid getting penalized for this. I learned this from self experience.