Yahoo has just done a mega-crawl

gilkesy · #1 (**permalink**) 09-02-2004, 09:38 AM

Hi all

Just looking at my site stats and I've seen that Yahoo has gone crazy crawling my site. Normally it visits around 30 times a day.

Yet today so far it has visited my site 700 times.

Any ideas as to why?

jestep · #2 (**permalink**) 09-02-2004, 09:47 AM

Normal Yahoo traffic on my sites. MSN is going crazy again though.

sfowler · #3 (**permalink**) 09-02-2004, 10:47 AM

Same for me with msn. Has hardly left the last three days.

gilkesy · #4 (**permalink**) 09-02-2004, 12:55 PM

Well, according to the stats program its Yahoo, but the actual crawler is from Inktomi Corporation. Does that mean this is MSN too?

robinev · #5 (**permalink**) 09-02-2004, 04:43 PM

Inktomi is now part of Yahoo. It's one of several companies that Yahoo bought out so they could market their own search engine.

ronniethedodger · #6 (**permalink**) 09-03-2004, 04:36 PM

Quote:

Originally Posted by gilkesy

Well, according to the stats program its Yahoo, but the actual crawler is from Inktomi Corporation. Does that mean this is MSN too?

Yahoo has not updated the designation on that spider. If you look at your server logs, you will see that the User-agent field clearly identifies it as Slurp.

Yes, those results show up at MSN Search ... at least until MSN rolls out their new search product which is supposed to happen before the year is out. Well, according to Gates it is supposed to.

zoreli · #7 (**permalink**) 09-08-2004, 06:33 AM

I notice that some of my new sites not listed on yahoo until now, were in yahoo for a few days, and now they are gone again. My guess is that update is in progress.

No changes on MSN for the moment. I guess that MSN will wait update to be done completly.

I hope to see my new sites listed in yahoo next week.

sfowler · #8 (**permalink**) 09-10-2004, 12:47 PM

Yes, slurp had a real good wallow in my site as well yesterday.

SioComms · #9 (**permalink**) 09-14-2004, 07:19 PM

I have a hits tracker proggy coded on my web pages and can tell what link people came in from on the search engines from and the keywords they used but how do I tell if it is a spider visiting the site?

Forgive me as am tweaking a lot of my pages just now and finding out new stuff all the time.

regards

Terry

ronniethedodger · #10 (**permalink**) 09-14-2004, 08:23 PM

SioComms - You can use a server log analyzer such as AWstats (OpenSource) or Sawmill (commercial). You can actually get Sawmill free, by doing testing for them. Details here.

SioComms · #11 (**permalink**) 09-14-2004, 11:06 PM

Thanks for that, have downloaded it and will try it out, now I can access my webserver stats remotely from my webspace provider so the CGi script one is the one to use?

never had to use one of these programmes before, I take it I can crunch all the data into this programme and it will sort out the info?

any tips apprieciated.

Regards

Terry

ronniethedodger · #12 (**permalink**) 09-15-2004, 01:11 AM

Quote:

Originally Posted by SioComms

Thanks for that, have downloaded it and will try it out, now I can access my webserver stats remotely from my webspace provider so the CGi script one is the one to use?

I take it you are testing out Sawmill. They have two ways to run it ... either as a desktop application or install it on your server.

I suggest running the exe to get a feel for it first. The Sawmill site has tons of help files on how to work this thing -- just play around with it. Not the hard to figure out.

SioComms · #13 (**permalink**) 09-16-2004, 09:17 PM

Thanks Ronnie, will have to play with it desktop side as my business server host won't let me run PHp applications on the server (one of the reasons am changing hosts at end of year).

Thanks for your help

Regards

Terry