iEntry 10th Anniversary Forum Rules Search
WebProWorld
Register FAQ Calendar Mark Forums Read
Yahoo! Discussion Forum Yahoo Search discussion. Any topic or subject specific to Yahoo should go here. You will also find a subforum dedicated to YPN & Panama.

Share Thread: & Tags

Share Thread:

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 09-02-2004, 10:38 AM
WebProWorld Member
 
Join Date: Mar 2004
Posts: 55
gilkesy RepRank 0
Default Yahoo has just done a mega-crawl

Hi all

Just looking at my site stats and I've seen that Yahoo has gone crazy crawling my site. Normally it visits around 30 times a day.

Yet today so far it has visited my site 700 times.

Any ideas as to why?
Reply With Quote
  #2 (permalink)  
Old 09-02-2004, 10:47 AM
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: May 2004
Location: Austin, TX
Posts: 1,229
jestep RepRank 0
Default

Normal Yahoo traffic on my sites. MSN is going crazy again though.
Reply With Quote
  #3 (permalink)  
Old 09-02-2004, 11:47 AM
sfowler's Avatar
WebProWorld Veteran
 
Join Date: May 2004
Location: Vienna, Austria
Posts: 953
sfowler RepRank 0
Default

Same for me with msn. Has hardly left the last three days.
__________________
Simon - Biogas, Rauchgasanalysator and Rauchgasanalysegerät
Reply With Quote
  #4 (permalink)  
Old 09-02-2004, 01:55 PM
WebProWorld Member
 
Join Date: Mar 2004
Posts: 55
gilkesy RepRank 0
Default

Well, according to the stats program its Yahoo, but the actual crawler is from Inktomi Corporation. Does that mean this is MSN too?
Reply With Quote
  #5 (permalink)  
Old 09-02-2004, 05:43 PM
WebProWorld Member
 
Join Date: Jun 2004
Location: Seattle WA USA
Posts: 86
robinev RepRank 0
Default

Inktomi is now part of Yahoo. It's one of several companies that Yahoo bought out so they could market their own search engine.
Reply With Quote
  #6 (permalink)  
Old 09-03-2004, 05:36 PM
ronniethedodger's Avatar
WebProWorld 1,000+ Club
 
Join Date: Aug 2003
Location: Central US
Posts: 1,265
ronniethedodger RepRank 1
Default

Quote:
Originally Posted by gilkesy
Well, according to the stats program its Yahoo, but the actual crawler is from Inktomi Corporation. Does that mean this is MSN too?
Yahoo has not updated the designation on that spider. If you look at your server logs, you will see that the User-agent field clearly identifies it as Slurp.

Yes, those results show up at MSN Search ... at least until MSN rolls out their new search product which is supposed to happen before the year is out. Well, according to Gates it is supposed to.
Reply With Quote
  #7 (permalink)  
Old 09-08-2004, 07:33 AM
WebProWorld Member
 
Join Date: Jul 2004
Location: Greece
Posts: 68
zoreli RepRank 0
Default

I notice that some of my new sites not listed on yahoo until now, were in yahoo for a few days, and now they are gone again. My guess is that update is in progress.

No changes on MSN for the moment. I guess that MSN will wait update to be done completly.

I hope to see my new sites listed in yahoo next week.
Reply With Quote
  #8 (permalink)  
Old 09-10-2004, 01:47 PM
sfowler's Avatar
WebProWorld Veteran
 
Join Date: May 2004
Location: Vienna, Austria
Posts: 953
sfowler RepRank 0
Default

Yes, slurp had a real good wallow in my site as well yesterday.
__________________
Simon - Biogas, Rauchgasanalysator and Rauchgasanalysegerät
Reply With Quote
  #9 (permalink)  
Old 09-14-2004, 08:19 PM
WebProWorld New Member
 
Join Date: Sep 2004
Location: Scotland, UK
Posts: 3
SioComms RepRank 0
Default

I have a hits tracker proggy coded on my web pages and can tell what link people came in from on the search engines from and the keywords they used but how do I tell if it is a spider visiting the site?

Forgive me as am tweaking a lot of my pages just now and finding out new stuff all the time.

regards

Terry
Reply With Quote
  #10 (permalink)  
Old 09-14-2004, 09:23 PM
ronniethedodger's Avatar
WebProWorld 1,000+ Club
 
Join Date: Aug 2003
Location: Central US
Posts: 1,265
ronniethedodger RepRank 1
Default

SioComms - You can use a server log analyzer such as AWstats (OpenSource) or Sawmill (commercial). You can actually get Sawmill free, by doing testing for them. Details here.
Reply With Quote
  #11 (permalink)  
Old 09-15-2004, 12:06 AM
WebProWorld New Member
 
Join Date: Sep 2004
Location: Scotland, UK
Posts: 3
SioComms RepRank 0
Default

Thanks for that, have downloaded it and will try it out, now I can access my webserver stats remotely from my webspace provider so the CGi script one is the one to use?

never had to use one of these programmes before, I take it I can crunch all the data into this programme and it will sort out the info?

any tips apprieciated.

Regards

Terry
Reply With Quote
  #12 (permalink)  
Old 09-15-2004, 02:11 AM
ronniethedodger's Avatar
WebProWorld 1,000+ Club
 
Join Date: Aug 2003
Location: Central US
Posts: 1,265
ronniethedodger RepRank 1
Default

Quote:
Originally Posted by SioComms
Thanks for that, have downloaded it and will try it out, now I can access my webserver stats remotely from my webspace provider so the CGi script one is the one to use?
I take it you are testing out Sawmill. They have two ways to run it ... either as a desktop application or install it on your server.

I suggest running the exe to get a feel for it first. The Sawmill site has tons of help files on how to work this thing -- just play around with it. Not the hard to figure out.
Reply With Quote
  #13 (permalink)  
Old 09-16-2004, 10:17 PM
WebProWorld New Member
 
Join Date: Sep 2004
Location: Scotland, UK
Posts: 3
SioComms RepRank 0
Default

Thanks Ronnie, will have to play with it desktop side as my business server host won't let me run PHp applications on the server (one of the reasons am changing hosts at end of year).

Thanks for your help

Regards

Terry
Reply With Quote
Reply

  WebProWorld > Search Engines > Yahoo! Discussion Forum

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 06:32 PM.



Search Engine Optimization by vBSEO 3.3.0