Please help interpret server stats!

[carol]

I'm not sure if this is the correct forum for this questions - apologies in advance if not. But anyone who could help here would be most appreciated!

My server stats (Live Stats, provided by hosting company) is showing that the most commonly accessed page on my site is my 404.htm page, and it is also the top entry point page. (Not that I get a huge amount of traffic, but this was 30% of it last month!) I'm kind of at a loss to understand this. What it tells me is that some external site is pointing people to a page that doesn't exist, but how do I figure out what external site, and what page, so I can do something about it?

Thanks for any input,
carol

[cyanide]

I'm not familiar with that program.

But does it tell you where the links are coming from?
or the page people are trying to access?
or keywords?, then you can go to google and track down the page.

Are you using a custom error404 page?
If not you should. It will atleast help keep people on your site.

[carol]

Thanks for your response cyanide. Yes, I'm using my own custom 404 page. But the Live Stats doesn't tell me where the links are coming from - that's the problem. Or, maybe it can be configured to do so, but I don't know how. I can set up "watches" if I know the incoming URL - kind of a catch 22. I'm at a loss!!

carol

[publicgadfly]

I don't know the answer to your question, BUT
change your 404 page into a real page :-)
Yep- put all the info from your "index" page there. That way those mis-directs don't get lost and they serve a positive purpose.

[carol]

Thanks - that's worth considering!!

carol

[minstrel]

I don't know the answer to your question, BUT change your 404 page into a real page. Yep - put all the info from your "index" page there. That way those mis-directs don't get lost and they serve a positive purpose.
I agree to a point, but I would suggest you don't duplicate your "home page" for several reasons, not the least of which is that if you are getting that many 404 hits you want some way to make it very clear to people and spiders that whatever they're trying to find doesn't exist.

What I do on my 404 page is, first, label it clearly as a "page not found BUT..." page. Then, I provide links back to the main site index, sitemap, home page, etc., but I also include a site search box in case the visitor was directed there from a search engine using a specific keyword search - if they type that word into the search box on my page and it doesn't go anywhere, my logs will tell me what the word is and I can make adjustments to how my site is being indexed locally. (You can see my 404 page here (http://www.psychlinks.ca/error.htm) if you are interested.

Another approach which has been suggested in these forums is to direct 404s to your sitemap page - not a bad idea but, again, I think the drawback to this is that you may not be making the 404 obvious enough to correct those errors.

You may be able to look at your logs to see what keywords or keyphrases are getting people to your site and get some clues there as well - if it's a keyword that doesn't match any page or content on your site and you wish it did, you may be able to add something... Look also at "top referrers" and see if it's a bad link on one of your other pages - you might check this independently with a link checker (I use the freeware program Xenu Link Checker to do this).

[rlrouse]

One day I was going through my error logs and noticed that quite a few users were incorrectly typing in a URL for one of my pages (a very common misuse of a synonym). Of course they ended up at my custom error page which I link to my sitemap.

I created a new page with the incorrectly typed page name and different (but relevant) content. That page now gets an average of 125-150 unique visitors per day.

Your error and referral logs are full of gold if you know how to mine it.

[minstrel]

One day I was going through my error logs and noticed that quite a few users were incorrectly typing in a URL for one of my pages (a very common misuse of a synonym). I created a new page with the incorrectly typed page name and different (but relevant) content. That page now gets an average of 125-150 unique visitors per day.
I never thought of doing that, I admit, although I do include misspellings in keywords... not a bad idea, although again my preference would be to have a page that tells people about the error and redirects them...


Your error and referral logs are full of gold if you know how to mine it.
Absolutely... it's an excellent way of finding out how people are trying to find you and how to make it easier for them to do that.

[carol]

minstrel - thanks for the advice about putting a site search box on my 404 page. Strangely enough, I've done that for a client, but it somehow didn't occur to me to put one on my own site! LOL!


rlrouse wrote: Your error and referral logs are full of gold if you know how to mine it.
Sorry, I guess I didn't word my original question clearly enough. That's precisely what I need help with. I will have to call my hosting company (who is providing the server stats) and ask them for assistance.

Thanks to both of you,
carol

[rlrouse]

Hi Carol,

Check your control panel. Most have direct links to your stats and log files.

[carol]

Yes, I can reach the stats through the control panel. The stats are showing that the most commonly accessed page on my site is my 404.htm page, and it is also the top entry point page. But it doesn't show me who or what is directing the traffic there - at least, I can't figure that out. I can see only the top 10 referrers, which includes Google, but doesn't say what keyword referred them there. I'm looking for advice as to how to mine the gold and leave the ore (sorry if this is a poor reference to your analogy...)
Thanks,
carol

[rlrouse]

Which stats application does your web host have set up for you? Urchin and AWSTATS are both very good, among others. Many hosting companies will have two or more set up in the control panel so you can choose (or use both).

Try contacting your web host and ask them to tell you the simplest way to view your log files (or better yet, download them into a spreadsheet).

[minstrel]

Carol:

First, see my advice above about running a link-checker on your site - with that many 404 errors, my first guess would be it's coming from another page on your site where you've mispelled a URL or improperly parsed a relative URL.

If that comes up clean, and your Host isn't helpful regarding extended stats, you might try using the HTTP_REFERER tag on the custom error page - it's been a long time since I tried to use this so please... anyone else feel free to jump in here - see HTTP1.1 Header Field Definitions (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html):



14.36 Referer
The Referer[sic] request-header field allows the client to specify, for the server's benefit, the address (URI) of the resource from which the Request-URI was obtained (the "referrer", although the header field is misspelled.) The Referer request-header allows a server to generate lists of back-links to resources for interest, logging, optimized caching, etc. It also allows obsolete or mistyped links to be traced for maintenance. The Referer field MUST NOT be sent if the Request-URI was obtained from a source that does not have its own URI, such as input from the user keyboard.

Referer = "Referer" ":" ( absoluteURI | relativeURI )

Example:

Referer: http://www.w3.org/hypertext/DataSources/Overview.html

If the field value is a relative URI, it SHOULD be interpreted relative to the Request-URI. The URI MUST NOT include a fragment. See section 15.1.3 for security considerations.


On the other hand, certain server features, proxies, or firewalls can block this header:


The ISP you are using may have a proxy server which is blocking the HTTP_referer tag. This referrer tag enables the message system to track what submission you came from. If it is blocked it will take you to an error page. Please contact your ISP and report the problem.


Internet proxy servers and traffic filters. Programs such as Ad-Blocker, Webwasher and Norton Internet Security and some HTTP proxies filter out HTTP-REFERER information, you must disable such software, or configure them to not filter out HTTP-REFERER information.

Now, as I said, I played with this tag several years ago so your post triggered a vague memory and sent me looking for info, but I don't pretend to understand what it can and cannot do - perhaps someone else here at WPW can either clarify this or post a message saying, "Pssst! did you see this? Minstrel has finally lost it - that's right - no contact with reality left at all!".

But in any case, start with the "erroneous link from another page on your site" hypothesis...

[carol]

You guys are good! Actually I have checked the site with a link checker - Web Link Validator kept saying everything was ok. But when I used Xenu, it showed several image files as generating 404s. Now, I could see the images (I was addressing them as "../portfolio/whatever.gif") so it didn't occur to me that they couldn't be found?? But when I changed that to "portfolio/whatever.gif" the errors evaporated. So I think this particular problem is solved.

(Sorry you went through all the mental gymnastics for the HTTP_REFERER thing...)

Of course, I should also take the advice and talk with my hosting company about downloading the raw stats.

Thanks so much!
carol