What can you do about who links to your site?

It seems to me there is no way I can stop (let alone even track down) anyone who wants to link to my sites!

Using this as a factor should put google on the last possible ranking for a search such as 'search engine' since so many crappy sites link to them. This can not possibly be a serious factor in ranking now... come on...