What is going on? - Radical Stats this month!

greeneagle · #1 (**permalink**) 01-06-2004, 10:09 AM

What the heck is going on this month?
Stats are very strange.
Here is latest bot visits for first full 4 days in Jan 04:

For the last 3 months LookSmart has been increasing crawl frequency and depth! So far this month they are outpacing GOOGLE for the first time! What is this all about? Do they not query the server for updated files as adequately as GOOGLE, or are they hitting the relevancy highway?
What in the heck is going on?
Who is this unidentified "Crawl" bot (not resolved by AW stats) but acts like a major bot?
Who is this unidentified "Robot" that AW stats can't resolve?
Both are with me every month and have the characteristic signature of major search engine bots.
Can one or both be bot developments in BETA for major search engines? Is one the rumored new bot for MSNSearch?
Someone's powerful phone browser bot?

Here are my new first full 4 day Jan04 stats on Browsers:

PDA phone browsers at 52%! What the heck is going on? --- never had more than 26% or so!
I'll have 20 Meg of bot crawls this month again, and I have an extensive robots.txt file:
http://www.mountaineagleweb.com/robots.txt
It's going to be a wild year!
If anyone can shed light ---- Please do!
Thanks
Ken

achronister · #2 (**permalink**) 01-06-2004, 11:26 AM

greeneagle,

You're right, those are some strange stats. My guesses (and they're only guesses, someone with a little more bot experience may be a better resource) but the unidentified bot may be some spam crawler searching your site for email addresses. I get that a lot. Looksmart is increasing thier crawls on my site as well, not sure why but it's fine with me. My site updates daily. If yours updates often the last update may be the answer.

52% PDA traffic?? I've never seen that. Maybe a bunch of your friends got those new Sprint Vision phones that I'd like to throw out my window and are just testing it out on your site. I have really no explination on that one. There are a few others on here that may be able to get that one.

Hope that helps a little anyway.

Aaron

greeneagle · #3 (**permalink**) 01-06-2004, 01:58 PM

Thanks achronister,

The two search engines in questions are not known spam harvesters and they have been with me a long time. They act more like a search engine bots. It's not likely with their crawling characteristics and as long as they have been around that they would not be listed in robots.txt exclude recommendations on the Internet, unless I missed it somewhere.

The PDA may level out to 20-27% as normal for my site by the end of the month, but I have never seen it start that high. What's amazing is that I have 750 pixel wide graphics on my main pages - but it is not so PDA limiting on my webmaster resources pages.

No, I don't know a single person that uses PDA phone browsers. In fact, I have never even seen what my site looks like on one. I have the capability on my own Sprint phone and never saw the use in the extra charges.

Ken

Dragonsi · #4 (**permalink**) 01-06-2004, 03:26 PM

It seams general if I am getting crawled by looksmart amoung other bots (including unknown) - most of my sites have quite a low PR (I'm working on it), so this must be some general new order set for the new year.

What is afoot?

greeneagle · #5 (**permalink**) 01-06-2004, 04:28 PM

A PR3 is not too bad -- you'll get there if yiu are working on it!
Ken

ronniethedodger · #6 (**permalink**) 01-06-2004, 05:01 PM

I do not find the percentage applicable. For one it is too early to tell what that is going to level out with over the next 30 days. Plus you are comparing it with one of the more unusual months (trafficwise) of the year, and that being the month of December.

I know I raised this question once before when you brought it up last month. Can you extract from your log files which pages the PDA referrals are entering at? If you can, then it will narrow down what these referrals are looking for a little bit easier.

If you determine the entry point of the referrals, you may find that it is some obscure page that is popping up in their search engines that the PDA's use. If it is your home page though, then that would be a different story.

It would also be nice to know what engine these PDA's are actually using too. See what is setting them off about your site, then use that to your advantage. As I also said before, you have yourself some untapped traffic here -- and I am still hoping that one of your 2004 predictions will still come true (if you know what I mean...hehehehe) ;0)

greeneagle · #7 (**permalink**) 01-06-2004, 05:20 PM

Ron,

If there are any AW stats (except keywords and keyphrases) I can post that would give you a clue on how I am getting this kind of phone browser traffic, you let me know, I'll digitize it in a small jpg and post it! ---- I don't have a clue --- Completely lost!

Ken

greeneagle · #8 (**permalink**) 01-06-2004, 05:31 PM

achronister said:
"...Looksmart is increasing thier crawls on my site as well, not sure why but it's fine with me..."

Is everyone else seeing the increased LookSmart crawl too?

Ken

minstrel · #9 (**permalink**) 01-07-2004, 02:36 AM

No need to worry - the "unknown bot" stats don't mean anything nefarious. This ia "faeture" of awstats.

Do a search for awstats on the WebProWorld forums - I and others posted about this previously but I'm too tired tonight to look - or use Google Groups. Either way, what you'll find is that awstats is known to default to that rather cloak-and-dagger term for bots not in its "database", and that includes a couple of well-known search engines.

ronniethedodger · #10 (**permalink**) 01-07-2004, 03:10 AM

Quote:

Originally Posted by greeneagle

If there are any AW stats (except keywords and keyphrases) I can post that would give you a clue on how I am getting this kind of phone browser traffic, you let me know, I'll digitize it in a small jpg and post it! ---- I don't have a clue --- Completely lost!

I am not sure how it is done in AWStats. I know how to do it in Sawmill.

The basic premise is that in your server log file each entry has information on (1)Browser Type and (2)What Page Was Requested.

In Sawmill, I can filter my results and display any part of those results. So I would filter for PDA browsers only, then only display the pages they requested. Just that easy.

Keywords would be the same way, except the process for cross-referencing to the page they entered on is tougher formula to set up and I am not sure I could even explain how to do it.

But you still could filter all the other browsers out, and see what keywords are hitting just for PDAs. It would also be nice to have a list of the SE's that the PDA's are being referred from.

All of these little bits of information put together will give you the whole story of how they are coming to your site and what for. Once you find that out, then you will see what you can do with it.

Right now....your site is not PDA compatible is it? Even if it were, there is no guarantee these PDA's are gonna find what they are really after. They may be looking for Standard Oil Gas Stations within a half mile of I-90 Exit 109, cuz they are all out of gas and only have a gas credit card.

One thing is for sure. Your stats are over inflated because of these PDAs. I would not consider them in your stats, much like I filter out worms from mine. Useless information.