Remnant tables left behind by deactivated/deleted plugins do no harm. If you delete a plugin from your Dashboard and it leaves tables behind, that is a sign of some sloppy authoring and probably a good thing you dumped it.
If you look at how tables are stored in MySQL, each has its own files for data and indexing separate from others. It is not like all the tables in a single database are stored in one huge file.
I feel plugins do cause a lot of load. Can try to disable them and try.