Spider Tracking/Analysis

**incrediblehelp** · #1 (**permalink**) 05-06-2008, 08:16 AM

Anyone know of a good third party tool that does good spider analysis and tracking? Google Analytics doesn't have any since it is JavaScript based. Click Tracks is limited an pretty poor in my opinion.

I would like to be able to view the spider activity of any of my URL's on my website over a long period of time. Seems simple enough but I have yet to find a tool that can help me with this.

kgun · #2 (**permalink**) 05-06-2008, 08:55 AM

What about CrawlTrack, the spiders and crawlers tracking script. ?

Is it possible to modify the script?

My idea if I had to do it myself:

On every page, you include (once?) a script that scan the page and put every URL in an array.

PHP Code:

var links_onpage1=document.getElementsByTagname('a');
Loop through every link in the array with a(n inbuilt) function like PHP: $_SERVER - Manual
Put a (unix?) time stamp on the result and store it in a mySQL database.

**incrediblehelp** · #3 (**permalink**) 05-06-2008, 09:27 AM

yeah I have used that one for PHP websites, but what about aspx websites?

kgun · #4 (**permalink**) 05-06-2008, 09:33 AM

I think part one in my do it yourself proposal, is independant of language, since

"The Document Object Model is a platform- and language-neutral interface that will allow programs and scripts to dynamically access and update the content, structure and style of documents. The document can be further processed and the results of that processing can be incorporated back into the presented page. This is an overview of DOM-related materials here at W3C and around the web".
Source: W3C Document Object Model

I have never used and know nothing about inbuilt functions in that language, so I can not help you more. May be you may write the function yourself.

**incrediblehelp** · #5 (**permalink**) 05-06-2008, 10:11 AM

Not looking to write it myself, but may have to. You would think some thing like this is already available.

kgun · #6 (**permalink**) 05-06-2008, 10:49 AM

If you have a problem in PHP, "99 % of them are solved already" if your are able to Google the right term. I don't know how it is in ASP.
I don't like to suggest another forum, but the best ASP forum I know of is the Norwegian W3Schools Forum (Powered by Invision Power Board) (look at the activity in the ASP sub forum there). You will soon discover that like many here they are very MS.