Spider Monitoring Tools??

[incrediblehelp]

What program or analytics tool does everyone use for monitoring you spider/robot traffic on your websites? Obviously using GA will not help you here and lots of other tools like AW Stats are not effective either because they just show you a global activity look. What tools is everyone using to help monitor robot traffic on page (URL) by page level?

I know WordPress users can use plugins like Crawl Rate Tracker, but hat happens when you want to analyze a regular website.

I know ClickTracks offers this, but it is to expensive for lots of clients.

Any suggestions?

[kgun]

Jaan, can't you use a panel on your site (hoster)? The quality of that tool may of course vary from hoster to hoster. It should also be possible to write a (php?) script to analyse the traffic in more detail. But since 99 % of all php programming needs are said to be solved already, there shoud be a php script that does the job if you use that language.

Googling:

php traffic analyse script

gives a lot of hits here.

You can vary that search to get more relevant hits.

The best solution may be integrated with a database, so you can make your own analytics software.

[incrediblehelp]

Yes I thought about building my own, but I wanted to see what was out there first before I spend time on someting that is already done.

cpanel only offers basic stiff like AW Stats and webalizer. Wont help for this need.

[yookoso]

I use a free program called CrawlTrack.

[incrediblehelp]

Originally Posted by yookoso

I use a free program called CrawlTrack.

Ah yes I forgot about that one. I will have to check that out as I have not used it in years.

[wpriley]

Look at this web analytics program:

Visitor Stats 2 - website statistics

It tracks and reports on both human visitors and spiders. It's a good (not great) stat package and, since it's hosted on your own server, costs only the $80 one time fee to license the software.