Google Blocks Scroogle

Garrett · #1 (**permalink**) 12-22-2003, 05:23 PM

Do you remember Scroogle? Daniel Brandt, of Google-Watch.org, created Scroogle when he discovered that you could see pre-update results for search terms when you included "+ jibberish term" after your key term.

His "scraper" queried Google for a term, then queried it again with the jibberish terms added so that you could compare your before and after-update search terms.

Blocked by Google. On December 7th, 11 days after he put up the Scroogle site, Google IP blocked his server so that he could no longer receive results.

He changed servers right away, and the same results continued to show until the 11th of December, when Google fixed the glitch that allowed the old results to show. He still has a list on his site of many of his old "before and after" results tests.

Why did Google block Scroogle? Google's explicit about their stance on third parties scraping their results: "You may not send automated queries of any sort to Google's system without express permission in advance from Google." (From their terms of service page.)

Daniel believes this rule is related more to automated queries that hit Google millions of times for spam or other nefarious purpose. At the peak of Scroogle's popularity it only hit Google about 20,000 times.

Plus, he originally built the scraper months ago to provide ad-free results. You can still search Google ad free from his site. Google took no notice of this scraper, which regularly requests results for 5,000 searches a day.

So why else would Google block Scroogle? Daniel thinks Google blocked his scrape because it showed Google's hand, something Google, as a highly secretive company, didn't like.

Scroogle was not intended to make money - there was no motive for Scroogle other than to comment on Google's latest update. Daniel believes that what he was doing was fair use, and that Google essentially violated his right to free speech.

But what was Google was so reluctant to reveal? According to Daniel, the hardest hit terms were those that included "your city" + "search term." Among these, realtors were especially hurt.

While they're mostly back to normal now (he did point out that, as of 12-22-03, "your city" + "hotel" still delivers irrelvant results), he thinks these changes point to an increased localization, a sort of Google Yellow Pages that will be an enormous money maker once Google goes public.

So, did Google block Scroogle because of a violation of company policy or because they didn't like that Scroogle revealed details about the change in their algorithm?

What do you think?

cbp · #2 (**permalink**) 12-22-2003, 08:48 PM

Google are well within their rights to ban Scroogle - they breached the Google terms of service due to the automated response - Googles ToS are explicit. Daniel Brandt can reach for all the conspiracy theories he likes - he has already expoused others.

As for the site, Scroogle - it was helpful. But to put it into context of its association with the Google-watch site, here is another wiew from the Watch-Google-watch site about this individual http://www.google-watch-watch.org/

CBP

Daniel Brandt · #3 (**permalink**) 12-23-2003, 01:35 PM

Scroogle wasn't an automated operation. Every search done by Scroogle was the result of search terms entered manually by visitors to the site. If I violated the terms of service, it was due to the reformatting of Google's output after scraping the results.

All I did was invite visitors to Scroogle to try out the filter demo, and I recorded their search term along with the "casualty rate" number for that term. The "hit list" used to be a moving 24-hour window, and unfortunately I didn't start archiving the daily results until December 6. When Google patched the hole that made the pre-filter vs. post-filter comparison possible, the scraper couldn't do its work. It was retired on December 12. I have the searchable hit list results available from Dec. 6 through Dec. 12, and each of these seven days includes about 5,000 terms.

I now wish I had the previous two weeks of data, because starting on December 10, the data was "corrupted" by Google's patch.

The scraper is still working as of today by using a second trick that has yet to be patched by Google. My confidence level in this data is lower, so these results are not available in the same way that the hit list was available.

The other proxy at Google Watch, the one that doesn't do a pre-filter vs. post-filter comparison, has been up for 16 months. Google has never blocked that one. They've known about it the entire time.

I believe that Google did not want you to see what the Scroogle comparison was demonstrating. The easy way out was to block me. When I switched servers and started scraping Google from an IP that was on a different Class C than anything Google could easily research, they couldn't find me. The original block is still in place, and the new server is still working.

Once they realized that the block wouldn't do what they wanted, Google implemented the patch. It took several days for the patch to propagate throughout Google's network, and you can trace the daily progress of the numbers on the hit list.

The value of the demonstration is that it changed the nature of the questions that webmasters were asking. Without the demonstration, all you had was a bunch of semi-blind webmasters describing what happened to the one or two terms that they had been tracking for months. Hundreds of webmasters did this on thousands of forum postings. It made your head spin, and didn't really reveal much.

With the demonstration, you could grasp the big picture. I feel that the demo is a "fair use" social criticism of Google. The Google Watch site is sponsored by a nonprofit, tax-exempt public charity. There is no way that the demo can load Google's servers.

The bottom line is that Google blocked Scroogle because they'd rather go for the quick fix, as opposed to becoming socially responsible corporate citizens. Google owes all of us an explanation of what they're trying to do with this new filter.

Nargule · #4 (**permalink**) 12-23-2003, 01:44 PM

Quote:

Originally Posted by Garrett

Scroogle was not intended to make money - there was no motive for Scroogle other than to comment on Google's latest update. Daniel believes that what he was doing was fair use, and that Google essentially violated his right to free speech.

Give me a break! Google hasn't vilolated Daniel's or any other person's Constitutional rights. They are a search engine. They can run their business any way they want.

Quote:

While they're mostly back to normal now (he did point out that, as of 12-22-03, "your city" + "hotel" still delivers irrelvant results), he thinks these changes point to an increased localization, a sort of Google Yellow Pages that will be an enormous money maker once Google goes public.

I tried "jacksonville + hotel" and "jacksonville hotel" and got exactly what I expected. I'm not sure what happended yesterday (12-22-03), but I'm sure it is not some huge conspiracy to prevent me from finding a hotel.

Quote:

So, did Google block Scroogle because of a violation of company policy or because they didn't like that Scroogle revealed details about the change in their algorithm?

What do you think?

Like I say, Google can block whomever they like - especially when their ToS are violated.

cbp · #5 (**permalink**) 12-23-2003, 03:15 PM

Quote:

The bottom line is that Google blocked Scroogle because they'd rather go for the quick fix, as opposed to becoming socially responsible corporate citizens

Wrong. Google blocked Scroogle becasue it breached its gudielines.

Here is what Google say:

Quote:

Don't use unauthorized computer programs to submit pages, check rankings, etc

Scroogle used an unauthorised computer program to check ranking - PERIOD - it can not been spelled out more simply than that. No conspiracy ... just plain facts.

CBP

minstrel · #6 (**permalink**) 12-24-2003, 09:02 PM

Quote:

Originally Posted by Daniel Brandt

I believe that Google did not want you to see what the Scroogle comparison was demonstrating.

(snip)

The value of the demonstration is that it changed the nature of the questions that webmasters were asking. Without the demonstration, all you had was a bunch of semi-blind webmasters describing what happened to the one or two terms that they had been tracking for months. Hundreds of webmasters did this on thousands of forum postings. It made your head spin, and didn't really reveal much.... With the demonstration, you could grasp the big picture. I feel that the demo is a "fair use" social criticism of Google.

(snip)

The bottom line is that Google blocked Scroogle because they'd rather go for the quick fix, as opposed to becoming socially responsible corporate citizens. Google owes all of us an explanation of what they're trying to do with this new filter.

Google doesn't owe anybody an explanation of what they're doing with the search engine they own.

There is no need for conspiracy theories - there is no mystery about why Google is doing what they're doing - for some time now, many SEO "experts" have been doing their best to manipulate Google's results to afford an unfair advantage to their websites in Google's SE listings - Google over the past several weeks has been taking steps to make that more difficult.

Why would they explain to you or anyone else exactly how they're doing that? So the same SEO experts who manipulated the system previously can find out how to do it again? And when you try to expose the changes Google has making to those same SEO experts, why are you surprised for even a nanosecond that Google took steps to shut you out?

You have a right to express your opinion about Google and their business practices to as many people as are willing to listen. You even have the right, I guess, to try to "expose" what it is you feel Google is doing, either "right" or "wrong". But Google is a private company (and even if they become public they are entitled to protect at least some of their trade secrets) and they don't owe you or me or anyone an explanation of exactly what they're doing, especially since to provide us with such an explanation would make the whole exercise pointless.

cbp · #7 (**permalink**) 12-24-2003, 10:54 PM

Just came across this analysis from April 2003 of some of the other claims by Mr Brandt:
http://searchenginewatch.com/serepor...le.php/2175251

BTW - I do like scroogle

CBP

minstrel · #8 (**permalink**) 12-24-2003, 11:48 PM

Wow! That's quite an article, cbp. Look especially at Accusation 5: "Google hires spooks", and Accusation 7: "Google's cache copy is illegal".

I'm surprised that the Kennedys, Lee Harvey Oswald, and any one of a dozen others including everyone associated with the Roswell incident or crop circles weren't dragged into the indictment in #5 and #7 is almost too silly to even comment on.

I encourage you to go read the article and make up your own minds about it.

What struck me while reading the article, apart from how plain silly many of these "accusations" are, is that the criticisms Brandt reserves for Google could as easily and perhaps more accurately have been aimed at Alexa, Yahoo, DMOZ, and any number of other similar services - so why Google? Because Google is currently the biggest and most successful and Brandt says he thinks Google should be government regulated, in spite of his stated fear that one of the problems with Google is its susceptibility to government privacy violations.

Personally, I'm inclined to be a lot more worried about Daniel Brandt than I am about Google...

Daniel Brandt · #9 (**permalink**) 12-25-2003, 06:25 AM

Anti-spam is only one of several theories concerning the reason behind Google's filter. It's not even the most popular theory.

For every spammy site that properly dropped in the rankings, others rose to the top. And for every spammy site that dropped, more than one non-spammy "mom and pop" site in a niche business lost their ranking. That's why the anti-spam theory isn't very compelling. You have to spend several hours a day combing the Net for information on this, for the last five weeks, to appreciate the dimensions of what Google's done with this filter.

The leading theory is that Google wanted to increase their earnings figures just prior to an IPO by forcing ecommerce into Adwords, and felt that they could get away with what they've done. So far the U.S. major media have not picked up on this filter story they way that major British media have. Google may be correct in their assumptions about getting away with it.

The second most popular theory is that it's an over-optimization penalty. This is known as OOP. I'd have to say that it's a distant second. One reason why it's attractive is that there seems to be a random element in the filter behavior. Such an element would be very compelling as an antidote for spammy techniques. However, the random element also lends itself to the "Google in incompetent and/or broken" theory, so the OOP camp starts splitting off into the "broken" camp and has yet to emerge with a coherent explanation.

There are various Hilltop, stemming, CIRCA, authority site, and theme-targeted PageRank theories. These are popular with certain SEO pundits, because as you can well imagine, the OOP interpretation has them on the defensive. Much of this is fog, in my opinion, and none of it works very well when it comes to explaining the evidence. The CIRCA paper is worth reading, however, because it's definitely the technology behind Adsense, and Google takes this technology seriously.

The theory that interests me at the moment is the geotargeting theory. After reading Google's FAQ on the option for regional targeting of Adwords, and combining it with what I've picked up from the CIRCA paper, I'm wondering whether Google has plans to roll out some geolocated features in the near future. Froogle is only for items that can be shipped. There's a huge amount of advertising dollars that can be collected from local services such as help wanted, real estate, travel facilities, auto sales and service, medical and legal professional services, etc. Google is experimenting with IP delivery to various marketing regions within the U.S. If Google is in a position to sell ads based on regional targeting, it would keep them competitive well into the future. As you can see, this theory is close to the IPO theory, in that it assumes that Google is trying to force ecommerce out of the organic results, and into pay for inclusion or pay for placement.

Another theory is that Google has to abandon PageRank because the patent is owned by the Stanford trustees, and Google want's to be unencumbered prior to filing for an IPO. I don't find this theory convincing.

So you have all these theories, and Google has no comment on any of this. At the same time, Google has considerable power over many lives. Many mom and pop sites that lost their rankings cannot afford to bid for Adwords, and I don't have to convince them that Google has power over their lives. All they can get from Google is an autobot email response.

It's a principle of democracy that one person's privacy ends at the point where that person's power affects the lives of others. This is so fundamental that I shouldn't have to be offering this as a civic lesson to anyone who graduated from junior high school. Whether you're a private corporation, a public corporation, an individual, or a Martian, it makes no difference. There are anti-trust laws, monopoly regulations, and restraint of trade considerations. The Federal Trade Commission has an interest due to consumer protection laws. The Federal Communications Commission would have an interest if they determined that Google was the equivalent of a "common carrier." There are many examples I could cite. The only question is whether those whom Google has impacted are sufficiently organized to press their case. The answer is obviously "no, they aren't." That's because Google is only six years old, and the regulatory environment took decades to evolve. Internet regulation is only beginning to evolve.

That's the political reality, but I'm arguing that we have the social and ethical right, as world citizens, to ask that Google explain themselves. Five or ten years from now, the political reality will have shifted and this argument from me won't be inviting such short-sighted responses. My main problem is that I'm ahead of my time.

But I must say, I didn't expect to be criticized on the grounds that no one even has the right to investigate Google. That's a real surprise to me.

minstrel · #10 (**permalink**) 12-25-2003, 10:10 AM

Quote:

Originally Posted by Daniel Brandt

Anti-spam is only one of several theories concerning the reason behind Google's filter. It's not even the most popular theory. For every spammy site that properly dropped in the rankings, others rose to the top. And for every spammy site that dropped, more than one non-spammy "mom and pop" site in a niche business lost their ranking. That's why the anti-spam theory isn't very compelling.

Where do you see any evidence that the filter is yet in its final form? that the changes initiated by Google have finished? Is there any chance that that could be the reason that things are still in flux?

Quote:

The leading theory is that Google wanted to increase their earnings figures just prior to an IPO by forcing ecommerce into Adwords, and felt that they could get away with what they've done. So far the U.S. major media have not picked up on this filter story they way that major British media have. Google may be correct in their assumptions about getting away with it. The second most popular theory is that it's an over-optimization penalty. This is known as OOP. I'd have to say that it's a distant second. One reason why it's attractive is that there seems to be a random element in the filter behavior. Such an element would be very compelling as an antidote for spammy techniques. However, the random element also lends itself to the "Google in incompetent and/or broken" theory, so the OOP camp starts splitting off into the "broken" camp and has yet to emerge with a coherent explanation.

I'm not going to even attempt a word-by-word rebuttal here. I'll just point out that I'm not convinced that the goals of the Google changes are at all random, nor that they are incompetent. I think all the real evidence is that the "OOP" theory as you call it is exactly what has prompted the changes. But I'm aware that that wouldn't provide much in the way of fodder for your website so I'm not very surprised that you find any alternative theory that hints at conspiracies and nefarious activities and even collusion with secret government agencies much more exciting than the truth, which is much more mundane - how many articles could you write and how many visitors could you get to visit your website and how many people would emerge to contribute to your organization if all you were discussing was Google's attempts to protect the integrity of its search engine?

Quote:

My main problem is that I'm ahead of my time.

Interestingly, this seems to be what all conspiracy theorists believe - other people just don't get it - I'm the only one who knows the truth.

Quote:

But I must say, I didn't expect to be criticized on the grounds that no one even has the right to investigate Google. That's a real surprise to me.

Can you show me where anyone said that? What I said was that you have every right to investigate whatever you wish, and to state your criticisms, etc., etc., but to expect Google to reveal everything about its search algorithms to you or to anyone else would be to destroy the integrity and therefore the utility of the search engine - so go ahead and demand that if you wish but please don't act surprised or cry "foul!" and "conspiracy!" when Google ignores you...

cbp · #11 (**permalink**) 12-25-2003, 05:11 PM

A lot of the conspiracy theory here is based on Google's IPO. Can anyone point me to a statement from Google that they are actually having on IPO?

I stand to be corrected, but I do not think that I have seen any such statement - there has obviously been a lot (actually more than a lot) of speculation that has the idea taken on a life of it own and become fact ... funny thing the internet :-)

BTW - I think they should have one and it probably makes business sense from Google's point of view

CBP

ronniethedodger · #12 (**permalink**) 12-25-2003, 11:45 PM

Quote:

Originally Posted by cbp

Can anyone point me to a statement from Google that they are actually having on IPO?

I tried doing a search at Google for the two-word search term "google ipo" and can't find a darn thing on it. Nothing really relative any way. ;0)

touristips · #13 (**permalink**) 12-26-2003, 05:55 AM

Am I wrong or is the florida dance a symtem of powers to be for a public offering.

To have a an IPO you must stress returns.

On a totally free enverion then results count.

By digesting a reply by Shari you should be buying clicks sounds like market is changing!!
Glad Shari made those comments since Martha Stewart is eating her words.

Let's us not focus on Shari since my internet hero Mr. Sullivan has flipped
(got paid) there is nothing wrong with that except his name is referenced.
I have not renewed S program opted for free version.
79 is a close combo to 86.

Why attempt to 86 you Dan.......simple I renewed and style froze my pc.
I don't eat cookies!
Then to find out that you sold it to someone else.
Wow!!!
There is nothing wrong with cashing out but having your name as a reference
makes no sense.

Google is going down....Intokmi has an option on yahoo & AOL which will evolve into better results.

All google can do is acknoledge the dance and go to pa format.

The fence between marketing and research is growing.

Since trashy dupicate results are the florida dance are in vogue seems marketing at google has breached that wall.

I didn't think google would want to dilute their true roots but now you want to monetize !!

Go public and cash out quick.
My brother does have a masters from Stanford.

The fence will kill Google name brand that was built for the right reasons.
I am not a rapper but Mr. google you'll never get a dollar out of me.

I doubt you'll get a buck from yahoo or aol since they have looked for alternative results.

Trying to be overture now with that fence building only will dilute what you had.
Hasta la Vista

janeth · #14 (**permalink**) 12-28-2003, 09:53 AM

Here is my two cents for what it is worth.

If I was Google and had gotten as big as they are, then I would look around.

I have MSN and Yahoo wanting a piece of the action. I'm as big as I will ever be, my job is just trying to stay #1 now.

I would try and make as much money as I could because no one stays #1 for ever and I can only get smaller as the others enter the game.

mikmik · #15 (**permalink**) 12-28-2003, 11:25 AM

Hi janeth
janeth wrote:

Quote:

Here is my two cents for what it is worth.

If I was Google and had gotten as big as they are, then I would look around.

I have MSN and Yahoo wanting a piece of the action. I'm as big as I will ever be, my job is just trying to stay #1 now.

I would try and make as much money as I could because no one stays #1 for ever and I can only get smaller as the others enter the game.

Yes! They have a legal 'right' to do business that way. That's what businesses do!
Daniel Brandt, with talk like this:

Quote:

That's the political reality, but I'm arguing that we have the social and ethical right, as world citizens, to ask that Google explain themselves. Five or ten years from now, the political reality will have shifted and this argument from me won't be inviting such short-sighted responses. My main problem is that I'm ahead of my time.

But I must say, I didn't expect to be criticized on the grounds that no one even has the right to investigate Google. That's a real surprise to me.

, you sound like the famous 'UFO Researcher' that was invited to a CSIOCOP convention (Commitee for Scientific Investigation OF Claims OF the Paranormal, a well known, well respected forum of scientific researchers and rational thinkers). He presented his subject which he had determined, after interviewing her about her reported 'alien abduction', that it was a genuine claim. When it was then revealed to him, in front of the assembly he was 'lecturing' that she was just making the story up as part of an investigation, and she herself stood there and told him herself, he them maintained, get this, that she had in fact really been abducted and didn't know it!!!!
I demand, as is my right, that you reveal to us your true motivation behind your obvious attempt at manipulating the media for attention. It is my fundamental right, as an investor to know if my decisions are being influenced by 'ulteriorly motivated' analysis. Furthermore, I will decide when you are telling the truth. It is my right to know. So let's have it. Do you follow your own rules?
You are not allowed to hide anything for your own well being, not if I am on to you for shenanigans.

So tell us, what are you really up to? You are trying to gain unfair advantage in the use of 'truth' in order to cover up your paranoia, so you have to now reveal all.

Yikes!
And quit reading my cerebral cortex with those transmissions!!

Oh ya.Touristips notes, for the record, thus:

Quote:

Go public and cash out quick.
My brother does have a masters from Stanford.

My mom does have a Doctorate of Psychology from the University of Alberta, and they are world renowned. That proves it.

ldyguique · #16 (**permalink**) 12-28-2003, 03:17 PM

I had just started coming to WPW when the November Google Dance when down. I'd never evem heard of the term, but the sheer angst of it was very real. It was a fascinating read, and I starting reading several threads on SEO -- grappling with new terminology and trying to make sense of a whole area of website development/maintenance that previously had been a total unknown to me. I had no point of comparison; so, I accepted all postings at face value and clicked on this link, ran this search and then that search. I even tried Scroogle.

As a long term Google searcher, I'd always revised my keywords to pull what was most relevant to me. If one set was too broad, I'd narrow it or define it more carefully. I didn't know that some of those ridiculous hits that I'd gotten on broad searches were "spammer" sites, as I'd only known the term spam as being used in terms of email. I just knew that I wasn't getting the results that I wanted and would revise my search accordingly.

I tracked down many a link or website from a poster and tried to determine "would I have searched on their choice of keywords?" No, I wouldn't have as they were either misspelled, irrelevant to "my" search, too broad (i.e., too many sites use exactly the same keywords,) or terminology that I don't use.

I observed the paranoia over "the Dance" vs the suspected upcoming IPO. It didn't have extraordinary impact on my thinking, but I tucked it away as a "possibility." I could tell that the timing of it, right before the Christmas season WAS affecting a lot of people.