Submit Your Article Forum Rules

Results 1 to 8 of 8

Thread: Greg Jarboe Talks Yahoo! Search

  1. #1
    Senior Member Garrett's Avatar
    Join Date
    Jun 2003
    Posts
    316

    Greg Jarboe Talks Yahoo! Search

    Yesterday I was fortunate enough to have lunch with Greg Jarboe, President and co-founder of SEO-PR, who shared some revealing insight into the new Yahoo! Search.

    Jarboe learned that the Yahoo! Slurp bot doesn’t follow links as closely as the Google bot. Instead, Slurp pays more attention to on page factors such as keywords and title tags to determine a site’s relevance.

    His theory: in an effort to get its own search solution up quickly, it’s possible that Yahoo! just rolled out the already-existing Inktomi technology with a few variations of its own. Within the next month or two, he predicts Yahoo! may also roll out AlltheWeb technology, which takes links into consideration the way Google does.

    Yahoo may be less inclined to roll out link analysis because link analysis concerns off page elements that are often out of the user’s control. These factors could potentially diminish your page rankings and you would not necessarily be able to control them at all.

    He also has one conjecture. “If you’re standing on the grassy knoll and you look West, where do you make the money? Through paid inclusion or through links?”

    Of course, the answer here is paid inclusion. He believes Yahoo! will hold off on the off page factors of AlltheWeb until it begins to get user complaints. This is just his conspiracy theory, however, and he has no evidence to support that statement.

    While paid inclusion offers an increased crawl rate of every 48 hours, it doesn't necessarily guarantee your position in the search engine results pages.

    Why participate in paid inclusion then? Jarboe is quick to point out that "Yahoo! is the tweaker's dream." With its frequent crawls, Yahoo’s paid inclusion program allows you to tweak on page elements like crazy to easily raise search engine rankings.

    That is going to be a major draw for advertisers to participate in the paid inclusion program.
    Garrett French
    Editor, WebProNews.com
    http://www.WebProNews.com

  2. #2
    I'm basically looking at Yahoo paid inclusing as Looksmart with a larger reach. The good'ol days of free Yahoo are over. Adapt to the changes and make it work for you, not against you.

  3. #3
    Senior Member
    Join Date
    Feb 2004
    Posts
    1,251
    Has anyone seen Yahoo! Slurp in their logs? Iget one that eats a lot but AwStats has it as "Unidentified".

  4. #4
    WebProWorld MVP minstrel's Avatar
    Join Date
    Jul 2003
    Location
    Ottawa, Canada
    Posts
    2,553
    Last I heard it was still identified as just "slurp"...

  5. #5
    WebProWorld MVP
    Join Date
    Jul 2003
    Posts
    1,933
    Quote Originally Posted by Dave Hawley
    Has anyone seen Yahoo! Slurp in their logs? Iget one that eats a lot but AwStats has it as "Unidentified".
    Aw stats is known to be a bit slow in updating their software, and I believe this is also dependant on the Host many of whom are not keen to disturb working systems with minor software updates. I have been seeing two unidentified spiders in AW stats,one of which I believe to be the MSN spider, and the other which has just hit my robots.txt 43 times in the past four days without visiting a page. This one may be Yahoo slurp.

    Robotstxt.org has a listing for the name MSNbot, but nothing listed for Yahoo slurp only Inktomi slurp.

  6. #6
    Member zenfort's Avatar
    Join Date
    Mar 2004
    Posts
    33
    Quote Originally Posted by Dave Hawley
    Has anyone seen Yahoo! Slurp in their logs? Iget one that eats a lot but AwStats has it as "Unidentified".
    Hi first tine posting.

    YET?!! I have had to htaccess 403 the critter already this month. the spiders 66.196.. have already eaten 400MB of bandwidth in the less than 4 days.
    I first had to cut off the spiders 66.196.. on the 26th of Feb. 2004. they had consumed 2GB of bandwidth since the 1st of Feb. Thay had been gobbleing more and more bandwidth every month for the past 56 months. First as Inktomi Slurp - now there are recognized as Yahoo Slurp.

    I am running PostNuke with PostCalendar and they will not obey however I try to talk to them in robots.txt.

    So I've adopted the tactic of forbidding them for periods of time - I guess.

    From Latest visitors:
    ===
    Host: 66.196.72.107 Url: /modules.php?op=modload&name=News&file=index&catid= &topic=10&POSTNUKESID=1e78c773952a8f6e889cfa85b37e ee44 Http Code : 403
    Date: Mar 05 07:41:08 Http Version: HTTP/1.0" Size in Bytes: -
    Referer: - Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

    Host: 66.196.72.45 Url: /delkergrid/techinfo.html Http Code : 403
    Date: Mar 05 07:40:42 Http Version: HTTP/1.0" Size in Bytes: -
    Referer: - Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
    ===

    And they even keep going after URLs that I haven't had on this server for years - the Delkergrid URL for instance of a few minutes ago. I know there are tons of links to the xenarts/xenonarts site cause it's been up for so long. I've been trying to transition the name to xenarts but I've about given up.
    namaste

  7. #7
    WebProWorld MVP minstrel's Avatar
    Join Date
    Jul 2003
    Location
    Ottawa, Canada
    Posts
    2,553
    Quote Originally Posted by Mel
    Quote Originally Posted by Dave Hawley
    Has anyone seen Yahoo! Slurp in their logs? Iget one that eats a lot but AwStats has it as "Unidentified".
    I have been seeing two unidentified spiders in AW stats,one of which I believe to be the MSN spider, and the other which has just hit my robots.txt 43 times in the past four days without visiting a page. This one may be Yahoo slurp. Robotstxt.org has a listing for the name MSNbot, but nothing listed for Yahoo slurp only Inktomi slurp.
    See above: at present "Inktomi slurp" IS "Yahoo slurp", at least as far as your web logs are concerned.

    See this thread:

    The instructions on the the Yahoo! Help page indicate that the 'bot is still called slurp:

    Yahoo! Slurp obeys the Robot Exclusion Standard. Specifically, Yahoo! Slurp adheres to the 1994 Robots Exclusion Standard (RES). Yahoo! Slurp will obey the first entry in the robots.txt file with a User-Agent containing "Slurp". If there is no such record, it will obey the first entry with a User-Agent of "*".

    Disallowed documents, including slash (the home page of the site), are not indexed, nor are links in those documents followed. Yahoo! Slurp does read the home page at each site and uses it internally, but if it is disallowed it is neither indexed nor followed. Example robots.txt:

    User-agent: Slurp
    Disallow: /cgi-bin/

  8. #8
    WebProWorld MVP
    Join Date
    Jul 2003
    Posts
    1,933
    Hmmmm..... well maybe Mistrel but then what are those things that identify themselves as yahoo! slurp in Zenforts Logs?

    From Latest visitors:
    ===
    Host: 66.196.72.107 Url: /modules.php?op=modload&name=News&file=index&catid= &topic=10&POSTNUKESID=1e78c773952a8f6e889cfa85b37e ee44 Http Code : 403
    Date: Mar 05 07:41:08 Http Version: HTTP/1.0" Size in Bytes: -
    Referer: - Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
    Host: 66.196.72.45 Url: /delkergrid/techinfo.html Http Code : 403
    Date: Mar 05 07:40:42 Http Version: HTTP/1.0" Size in Bytes: -
    Referer: - Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Similar Threads

  1. Microsoft & Yahoo In Talks Again!!
    By SemAdvance in forum Yahoo! Discussion Forum
    Replies: 6
    Last Post: 05-26-2008, 01:28 AM
  2. More on the End of Yahoo/AOL Talks
    By WPW_Feedbot in forum Search Engine Optimization Forum
    Replies: 0
    Last Post: 11-14-2005, 04:30 PM
  3. TiVo in talks with Google & Yahoo
    By ctabuk in forum Search Engine Optimization Forum
    Replies: 2
    Last Post: 04-25-2005, 10:49 AM
  4. SES Day 1: Yahoo Talks Personalized Search
    By Brittany in forum Insider Reports
    Replies: 24
    Last Post: 08-05-2004, 09:42 PM
  5. Yahoo! Talks BlackHat Optimization And Search As Media
    By Garrett in forum Insider Reports
    Replies: 13
    Last Post: 03-16-2004, 04:59 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •