Log Viewing Software, your take?

[nullvariable]

Hopefully this is a good place for this topic. It involves marketing so here goes.

I've started looking for a better log parsing/viewing software and I would like to know what everyone here is using. I've been using awstats and Do you use an 'offline' software and download your raw logs. Or do you use an online service that either a) creates its own logs using code or buttons or b) evaluates your raw server logs.

I personally shy away from anything that requires more (visible) code on my site as I feel it detracts from the professional look/feel. Plus you have less control over this code and can't fix it if it goes down. It also tends to be less accurate (not loading right everytime).

So that said does anyone have any products they recommend?

[brian.mark]

Sure. Take a look at ClickTracks, Urchin (now owned by Google), Webtrends, Index Tools or HBX (WebSideStory). The last 2 are page tagging only, so you'd continue to use awstats along with those to account for spiders.

Personally, I've been using Index Tools and find it to be worth every one of the hundreds of dollars per month we're spending. Their base package is reasonable ($19.95/month), but that's only 35,000 pageviews. We're using one of the upgraded versions and use the conversion analysis quite frequently to see the effects of our usability changes. Those packages start at $49.95/month including 100,000 pageviews.

Since changing from logfile to a tagging solution (Index Tools uses a 1x1 transparent image for those without JavaScript and tracks users without cookies by complex methods as well) I've enjoyed looking at our stats a lot more. I can see if we have any sort of a problem from any engine, do some ad-hoc reports ("what phrases convert into sales really well from MSN" or "how many visits does it take for someone coming from Yahoo to make a purchase") and really understand our visitors in ways that make sense to geeks and non-geeks alike.

Also, by having it hosted I'm able to set up reports that email themselves to people within our company based upon roles so we can get everyone working towards a common goal. Very, very useful stuff.

If you really like Logfile analysis, then Urchin has some cool features. It combines Logfile with tagging, but does it all through your server. I find the module pricing approach to be quite excessive, though. You can easily spend $5000 or more getting what you need.

Webtrends has been industry standard reporting for years, which is why so many others popped up seemingly overnight. They're a bit behind in some areas, but they do have conversion analysis, customized reporting (all data from your logfile is imported into a database for analysis) and most of the truly necessary features.

If you really want a cool view, ClickTracks and HBX both offer a page overlay option. This allows you to pull up your site and it overlays click percentages based upon visitors to your site clicking on to a link from that page. Very cool, but hard to really do something actionable with (unless they never see your checkout button or some other call to action - move it and see if it gets more clicks, that type of thing.) The reporting for HBX was astounding, but so was the pricetag when we looked at it. ClickTracks seems to be a decent comprimise, and they offer both tagged and logfile versions so you can see what they do.

Basically, all of the big names offer 15 or 30 day trials, so start downloading / implementing and see what really calls out to you. I've tried most of them, and for our business model I liked Index Tools. It also helped that the sales guy and I partied at the Googleplex together last year, but I wouldn't have signed up if I didn't think it was a working solution.

Brian.

[nullvariable]

Thanks for all the great information! I guess I am not convinced that a tagging solution is really necessary. I can generally discern which visitors went where on my sites. Granted I have fairly low volume traffic. But I just don't see how tagging pages could produce much additional information.

But hey that's me. I'm the kind of guy who will go and write half of the analysis program that he wants and then give up and use something else :-D

NV

[brian.mark]

Well, a few of the things that my tagging solution shows me that the log tools didn't are as follows:

1) Cookies enabled percentage
2) JavaScript enabled percentage
3) Visitors that purchased, what their search terms were, path through the site and how many times they've visited before.
4) Top search phrases by conversion rate
5) Top search phrases by revenue
6) Top search phrases by number of sales
7) Browser resolutions broken by percentages / visitors

Add in the "Up to the minute" reporting (I monitor during the day to make sure the conversion rates stay where they're supposed to be) and the live CPC analysis to get a very cool picture of what our marketing efforts are doing.

If you don't have much traffic, that's probably not necessary. We're trying to aggregate data for 100's of thousands of orders this year into meaningful reports, so it just isn't possible to look at the log files and figure out what works and what doesn't.

Brian.

[nullvariable]

Hmm, I see where you are coming from. And with thousands of things to track it does make sense to track conversions. Another question, did your company look at writing its own tracking software? The reason I ask is that it is pretty simple for me to right a little more code into my site to do the exact things you're mentioning with JavaScript and cookies and such. Were there cost advantages? I would think that paying a monthly fee for a service, while not a big deal for a larger company, would be much more costly in the long run. Also the log files should reveal the information about conversions and visitor paths, each entry in the log has the time and IP stamp which can be easily used for a more in-depth analysis.

I personally think that I will just write a simple data analysis program for the log files for now and any extra data that I feel I need I should be able to script into my existing code. For that matter I'll just give it to one of my programmers and see what he comes up with :-D Again tho, easier for a small company to do since we don't have as much traffic to deal with...


NV

[brian.mark]

Yes. Initially I wrote my own app. When our traffic spiked, our database server for tracking this was brought to its knees, making for a website that could barely function. We decided to have someone else host it at that point.

With millions of pageviews monthly, it just got to be more than what our Dual Xeon with 4GB of RAM could handle.

Brian.

[nullvariable]

Well if I had that many customers I guess it wouldn't be hard to justify :-D