I use G-site crawler, by far, the best and most complete program for your needs, actually it has so many features that you can get a little confused when you just start using it.