What program or analytics tool does everyone use for monitoring you spider/robot traffic on your websites? Obviously using GA will not help you here and lots of other tools like AW Stats are not effective either because they just show you a global activity look. What tools is everyone using to help monitor robot traffic on page (URL) by page level?
I know WordPress users can use plugins like
Crawl Rate Tracker, but hat happens when you want to analyze a regular website.
I know ClickTracks offers this, but it is to expensive for lots of clients.
Any suggestions?