For over a year now, Yahoo has spent a lot of money acquiring assets that will allow them to grab market share from Google. They bought Inktomi, which is feeding search results to the Microsoft network, and they bought Alltheweb, and AltaVista, and Overture. Last month they unveiled plans that will integrate these assets. The most controversial of these is Site Match, which involves embedding paid links into the main index. There has been very little comment about Site Match from the noncommercial sector. Almost all of the grassroots discussion revolves around whether the typical ecommerce webmaster can benefit from the Site Match model, or whether he would be better advised to rely on free crawling and indexing. The commentary from the pundits is even more shallow, and discusses whether Yahoo has the right stuff to crush Google. This article, on the other hand, looks at Site Match from the perspective of a nonprofit webmaster.

Site Match is clearly an attempt to monetize web search beyond the boundaries dictated by various relevancy algorithms. The first problem Yahoo faces when presenting Site Match is that they have to insist that paid listings will be treated the same as unpaid listings. This is one way that Yahoo can justify integrating these two into the main index. If Yahoo cannot successfully make this claim, then the FTC might be in a position to exercise their consumer protection mandate, and declare that Yahoo’s Site Match amounts to deceptive advertising.

Another way of satisfying the FTC would be to conspicuously label the paid listings so that consumers are properly informed. But the evidence suggests that Yahoo has no intention of doing this. The screen shot on the right is taken from Yahoo’s description of the difference between Pay-for-Performance and Site Match. The former is similar to Google’s AdWords program, where ads are labeled “sponsor results” and payment and position are determined by bidding. These are followed by algorithmic “web” results.

The clear implication is that one is paid and the other is unpaid. Google keeps the two types separate, and Microsoft hinted that their engine, due out in early 2005, may also keep them separate. Yahoo, on the other hand, plans to infiltrate Site Match into the algorithmic listings to make even more money. From this screen shot, and from the scuttlebutt on the forums, it appears that within the main index, the paid and unpaid results will all look the same to the consumer. The new format should become clearer after April 15.

Without labeling, Yahoo is compelled to emphasize that the Site Match listings will be ranked by relevancy exactly the same as the unpaid listings. I’m not convinced this is true. Google is not convinced this is true. Ask Jeeves is not convinced this is true. And Microsoft is not convinced this is true. Even if you think these last three are just tooting their own competitive horns, there are good reasons to be suspicious.

For one thing, Yahoo’s crawler is an underperformer. There are massive holes in Yahoo’s index. There are numerous examples of poor and erratic crawling by Slurp, their crawler, and assuming that the page was crawled to begin with, even worse indexing in Yahoo. This is true of nonprofit sites as well as many commercial sites. Yahoo guarantees a crawl every 48 hours for Site Match customers, using a dedicated crawler. Their poor unpaid crawling therefore provides an incentive to join Site Match, for those who can afford it. As long as this is the case, it also provides a disincentive for Yahoo to improve their unpaid crawling. Of course, in the long term Yahoo has to look like a real search engine, and eventually their unpaid crawling must improve.

Up to this point, there is no evidence that Yahoo is looking that far ahead, despite some rhetoric from Yahoo representatives on various forums. There have been some noises about Yahoo reaching out to acquire the “deep web” by absorbing databases from the Library of Congress and National Public Radio. Translated, this means that nonprofits might qualify for a free “trusted feed,” which would mitigate the poor crawling of the noncommercial sector. It will be interesting to see if Yahoo is simply spewing more rhetoric, or if they are serious about becoming a real search engine. So far Yahoo has not published any details or procedures for implementing this nonprofit trusted feed.

Yahoo is adding pay-per-click on top of their $49 annual fee for Site Match. The annual fee is per page, but drops to $29 each for two to ten URLs on a domain, and $10 for eleven or more. Each click is from fifteen cents to thirty cents. This is very expensive! Much of the discussion on ecommerce forums revolves around what sort of control webmasters will have over which clicks are charged to their account. There will be some geolocation control, but apparently no keyword control. Yahoo is touting the 48-hour crawl and the advanced reporting that come with Site Match listings.

The quick crawl is attractive to search engine optimizers, because they can get excellent feedback on their page tweaks, and adjust them to fit Yahoo’s algorithms. Although it requires some effort by the webmaster, this will, over time, almost certainly result in some ranking advantage for Site Match pages. It’s still rather expensive, however, and most commercial webmasters are taking a wait-and-see approach toward Site Match.

Ultimately, Yahoo’s claim that Site Match will not skew the rankings places the burden of proof squarely on Yahoo. The proxy on this site is tracking the pay-for-inclusion links that Yahoo ported from Inktomi. It shows that these links, for whatever reason, enjoy an average ranking advantage. If Yahoo insists on corrupting the main index with paid listings, then they should monitor the ranking differential, and impose a correction for Site Match rankings so that this differential approaches zero. Without such a system, Yahoo will have a tough time convincing me that Site Match has no influence on ranking.

The larger issue is why Yahoo feels the need to monetize the main index at all. This is a bad precedent, and one that Yahoo has not thought through very well. If Yahoo cannot make sufficient money on labeled listings riding on top of a pure unpaid index, then why assume that goosing the index with Site Match will yield even more golden eggs? There are some very fundamental contradictions in Yahoo’s approach. It looks like an approach designed by a loose committee of greedy MBAs eager for promotion, which was then imposed on vice-presidents hungry for the next tech bubble.

This week Microsoft’s Steve Ballmer announced that their new search engine will be launched in about a year. He also regrets that Microsoft waited so long to get started. What this means for Yahoo is that they have a captive platform for another year to help establish their reputation for web search. Until then, Microsoft is trapped between Yahoo’s Overture for ads, and Yahoo’s Inktomi for algorithmic search, and has nowhere else to go.

Clearly, Yahoo is in a turf war with Google. Who could want more than the combined platforms of Yahoo and MSN for a year, plus whatever remains of Alltheweb and AltaVista, to wage this war? Yahoo’s exposure is very high (about half of their total revenue is from search-related activities), and many of their cards are already face-up on the table. If they don’t show that they are committed to pure search excellence within the next twelve months, it will be impossible to recover.

Right now it looks like they have their priorities backwards. Yahoo is copying Google on the cheap, in an effort to rapidly expand their ad revenue. Instead, they should be committing more resources to noncommercial, informational, pure algorithmic search. The first step in this process would be to dump the entire Site Match program.

Daniel Brandt operates Public Information Research, PO Box 680635, San Antonio TX 78268-0635

Tel:210-509-3160

Fax:210-509-3161

Nonprofit publisher of NameBase, http://www.yahoo-watch.org, and http://www.google-watch.org/

namebase@earthlink.net