Google Says Blocking Countries Outside of the US is Against

[janeth]

This is a bit much.

Google says they will ban you if you block people from viewing your site.

Google Says Blocking Countries Outside of the US is Against Policies

If you are being spammed by a bunch of guys in India and would like to block the whole country, you better rethink.

[Tech Manager]

I suppose Google should stop helping China block content from outside the country.

I do a lot of work in regards to blocking traffic by country. So far I have seen absolutely no effect on any website taking appropriate steps to block an IP, an IP range or an entire country.

[janeth]

Originally Posted by Tech Manager

I suppose Google should stop helping China block content from outside the country.

I do a lot of work in regards to blocking traffic by country. So far I have seen absolutely no effect on any website taking appropriate steps to block an IP, an IP range or an entire country.

I agree and have had to do it for clients in the past. Sometimes it is the only way to save a site.

[crankydave]

Let's see... an ecommerce site that only services a particular region, regularly receives attempts of credit card fraud from outside that region, blocks those particular IP ranges, is going to be considered to be cloaking?

Is it April 1st or sheer lunacy?

Dave

[wige]

But think about this from Google's standpoint. As a search engine, Google can't tell what parts of the world you are blocking from accessing your site, at least reliably, and Google may not have a system in place to filter out your site from the results when people in those parts of the world run searches.

Google has stated repeatedly that their goal is to provide the best possible search results, and avoid users from clicking into an error message. If IP filtering is perceived by Google to make their results look faulty to those who are blocked, would you not expect them to take action, up to dropping the sites that do this?

[janeth]

Originally Posted by wige

But think about this from Google's standpoint. As a search engine, Google can't tell what parts of the world you are blocking from accessing your site, at least reliably, and Google may not have a system in place to filter out your site from the results when people in those parts of the world run searches.

Google has stated repeatedly that their goal is to provide the best possible search results, and avoid users from clicking into an error message. If IP filtering is perceived by Google to make their results look faulty to those who are blocked, would you not expect them to take action, up to dropping the sites that do this?

It's what I was thinking as well, but for the site owner it's a loose, loose situation.

[crankydave]

Originally Posted by wige

But think about this from Google's standpoint. As a search engine, Google can't tell what parts of the world you are blocking from accessing your site, at least reliably, and Google may not have a system in place to filter out your site from the results when people in those parts of the world run searches.

Google has stated repeatedly that their goal is to provide the best possible search results, and avoid users from clicking into an error message. If IP filtering is perceived by Google to make their results look faulty to those who are blocked, would you not expect them to take action, up to dropping the sites that do this?

I see, Google's business model can't "reliably" do something so the answer is to make it a violation of their guidelines to do so? That's rich.

Dear Google

These people keep coming to "my house" and stealing from me. Is it okay if keep them out of my house?

Dave

Dear Dave

No it is not. That would be evil. You have to let them in your house or we will punish you.

Google

[Tech Manager]

I'm beginning to wonder whether George Orwell wrote the rulebook for Google.

Google might want to be the best, most accurate Search Engine around, but they aren't going to stay their if they require people to dumbdown their security to accomodate Google results. If Google wants to penalize me for blocking Russia, China and Korea they can do so without penalizing websites with the rest of the globe.

[Tech Manager]

I haven't noticed anything in Googles rulebook that declares cloaking to include denying access to certain IP ranges.

Cloaking, sneaky Javascript redirects, and doorway pages

I suppose you could stretch the rule to suggest serving no content is the same as serving different content.

[Webnauts]

John Mu at Google said:

"Blocking all users outside of the US from being able to see your site would likely be considered cloaking and would be against our Webmaster Guidelines. Instead of blocking these users automatically, I would recommend that you add blocks based on the user's activity, not based on his location. "

And I think that is pretty fair.

[Tech Manager]

Originally Posted by Webnauts

John Mu at Google said:

"Blocking all users outside of the US from being able to see your site would likely be considered cloaking and would be against our Webmaster Guidelines. Instead of blocking these users automatically, I would recommend that you add blocks based on the user's activity, not based on his location. "

And I think that is pretty fair.

I have no sites that only allow USA access, but regardless, I think John Mu is entirely incorrect. Sure, blocking based on user activity might be preferable, but it can be impossible for some webmasters. I manage a significant number of servers and networks. Malicious traffic from Russia, China, North Korea and a myriad of other locales can be highly disruptive to email servers, websites, intranets, etc. Is it more expedient for me to spend my time actively responding to the behavior of traffic that is 99.9% malicious or is my time better spent blocking the traffic entirely?

As an IT Manager I don't plan security and protection of sensitive data around the whims of Google, Yahoo, MSN or others. I have an obligation to the clients I support to keep their sites, servers and traffic as safe as possible.

Strong Security is multifaceted. When it calls for blocking a range of IPs or an entire country I will do so.

Is Google going to penalize you for relying on SBL, PBL, XBL lists to identify IP addresses and IP ranges of spammers? Are they going to penalize you for blocking known IP ranges of comment spammers, brute force attackers and email harvesters?

Prevention is the first line of defense. If my security becomes solely based on behavior instead of proactive prevention, I have failed.

[crankydave]

Originally Posted by Webnauts

John Mu at Google said:

"Blocking all users outside of the US from being able to see your site would likely be considered cloaking and would be against our Webmaster Guidelines. Instead of blocking these users automatically, I would recommend that you add blocks based on the user's activity, not based on his location. "

And I think that is pretty fair.

Except that "users" are not assigned individual IP's. Blocking a particular IP or IP range prevents everyone in the "region" from access not just the individual.

Dave

[Tech Manager]

Excellent point Dave.

[Martinscholes]

If someone in -say- California attacks a website in Washington State but uses spoof ips of -say- Holland, then the website in Washington State would block Dutch ips, whereas the real villain is in California, laughing his socks off.

[Tech Manager]

A spoofed IP attack is not quite as simple as your post would lead someone to believe. IP spoofing doesn't truly hide your actual IP. Generally speaking, IP spoofing does not generally allow for a normal network connection. This type of activity generally entails a blind spoof.

While IP spoofing can happen it is more common to see various other attacks.

[Webnauts]

Originally Posted by crankydave

Except that "users" are not assigned individual IP's. Blocking a particular IP or IP range prevents everyone in the "region" from access not just the individual.

Dave

Already my team Information Architect is working on a script that blocks suspicious bots by IP for 24 hours, and then get unblocked. This thread is a good hint to extend that also for bad visitors. I will discuss that with him tomorrow to see what and how that would be possible.

[Tech Manager]

Originally Posted by Webnauts

Already my team Information Architect is working on a script that blocks suspicious bots by IP for 24 hours, and then get unblocked. This thread is a good hint to extend that also for bad visitors. I will discuss that with him tomorrow to see what and how that would be possible.

Let us know.

[Webnauts]

I was also thinking about something else.

We get in average 8-10 emails a day from Indian companies, offering links building, etc services, mentioning that they conform to the US CAN-SPAM law. We report all those mails as spam, since that law is not valid in Germany were we are located. In Germany unsolicited emails, no matter which nature, are strictly forbidden.

Now they began using our contact form. We created an IP based script which we wanted to add in our contact form, which would redirect all India visitors to our homepage instead.

Before we had the chance to go live, we read the thread at Googles Webmaster Help Center, which Janeth shared here too.

Looks like we only have two options:

1. Disallow our contact page in our robots.txt and implement our script.

or

2. Extend the script by blocking the transaction of the mail if the visitor comes from India.

In both cases it will not be cloaking. I assume the 2nd option would be possible and I obviously prefer, since I do not want my contact page to be de-indexed. I will check that with our IA too.

[Webnauts]

Originally Posted by Martinscholes

If someone in -say- California attacks a website in Washington State but uses spoof ips of -say- Holland, then the website in Washington State would block Dutch ips, whereas the real villain is in California, laughing his socks off.

This is what we also planning to implement: Http:BL Application Programming Interface (API) | Project Honey Pot
Worth to have a look.

Upon the chance, allow me to refer you all to the excellent service:
Distributed Spam Harvester Tracking Network | Project Honey Pot

[activeco]

Originally Posted by Tech Manager

IP spoofing doesn't truly hide your actual IP.

But a good proxy does and many backdoors are used exactly for that.
Martinscholes is right, most of the villain IP's are being intentionally chosen from another side of the globe.

[Webnauts]

Originally Posted by activeco

But a good proxy does and many backdoors are used exactly for that.
Martinscholes is right, most of the villain IP's are being intentionally chosen from another side of the globe.

This solution is not the best ever, but it still helps to block a respective number of proxy servers:

RewriteCond %{HTTP:VIA} !^$ [OR]
RewriteCond %{HTTP:FORWARDED} !^$ [OR]
RewriteCond %{HTTP:USERAGENT_VIA} !^$ [OR]
RewriteCond %{HTTP:X_FORWARDED_FOR} !^$ [OR]
RewriteCond %{HTTP:PROXY_CONNECTION} !^$ [OR]
RewriteCond %{HTTP:XPROXY_CONNECTION} !^$ [OR]
RewriteCond %{HTTP:HTTP_PC_REMOTE_ADDR} !^$ [OR]
RewriteCond %{HTTP:HTTP_CLIENT_IP} !^$
RewriteRule ^(.*)$ - [F]

[activeco]

Originally Posted by Webnauts

This solution is not the best ever, but it still helps to block a respective number of proxy servers:

No 'professional' proxy reveals itself in http headers.

[Feydakin]

Google India is more than welcome to drop my site from it's serps.. I don't sell to India.. Or Korea, or the Ukraine.. So all of those Google.s are more than welcome to drop me from their index.. It would save us a LOT of hassle on the customer service end answering emails trying to explain "why" we refuse to ship there..

[crankydave]

hmmmmmmmmmmm...

Why should I or any other website owner have to allow anyone or anything access to their "property" to be "compliant" with a set of "rules' or "guidelines" SPECIFICALLY designed to promote their own interests ESPECIALLY when they're stealing or trying to commit fraud?

Tell you what John... name a single IP you've blocked/redirected/anything that targets ONLY the individual "abusing" it and not everyone in that region on the same IP.

Dave

[Webnauts]

Originally Posted by crankydave

Tell you what John... name a single IP you've blocked/redirected/anything that targets ONLY the individual "abusing" it and not everyone in that region on the same IP.

I fully agree with that Dave. As you know I am an advocate of Web Content Accessibility, and if I would do otherwise as you said, I will be violating those standards.

[Tech Manager]

Originally Posted by activeco

But a good proxy does and many backdoors are used exactly for that.
Martinscholes is right, most of the villain IP's are being intentionally chosen from another side of the globe.

MartinScholes is semi-correct. Proxies are an all together issue and can be handled differently. Proxies exist all over the world. There are plenty of them right here in the United States. Many non-IT/network people tend to regurgitateless than accurate information and sometimes downright myths and urban legends about security.

Malcious traffic can be filtered in a multitude of ways with a wide variety of rulesets. Proxy detection filters can be added to weed out known and unknown proxies.

While proxies still pose a significant threat they are manageable.

Proxies make up a comparatively smaller portion of malicious/spam/hacking traffic than you might think. Currently, a much bigger problem is malcious Botnets. Botnets create a much more randomized assault as they rely on infected machines to spew their traffic.

I keep hearing the proxy argument position as why country or IP range blocks simply don't work. It's an internet urban legend.

The argument goes something like this, "If you block an IP range a smart hacker will just use a proxy to get around your protection; therefore it is better not do this."

This is circular reasononing. It is much like saying, "Don't bother putting a lock on your front door because an intruder will then use your backdoor."

Ok, so I'll put a lock on my backdoor too!

"No, don't bother putting a lock on your backdoor because an intruder will break in through one of your windows."

Well then, he's going to set off my silent alarm, alert my doberman and if he survives the dog he's going to be staring down a loaded gun. Problem solved.

As I've said many times, security is multifaceted. It starts with the concept of protection and permissions. Whom do you want to allow into your home, website, servers and networks? It then incorporates several layers of protection: hardware and software firewalls, blacklists and white lists (which can include network ranges and/or countries), proxy detection, XBL, PBL, XBL lists, traffic behavior patterns and response, secure coding, properly validated variables, bad-behavior-redirection, and myriad other proactive and reactive layers.

I think we need to be careful to avoid flippant comments about proxies and other "origins" of traffic. Unless we are accurate we tend to promote the myth more than the reality.

[janeth]

Originally Posted by crankydave

hmmmmmmmmmmm...

Why should I or any other website owner have to allow anyone or anything access to their "property" to be "compliant" with a set of "rules' or "guidelines" SPECIFICALLY designed to promote their own interests ESPECIALLY when they're stealing or trying to commit fraud?

And why should Google have to allow anyone on to their property that causes them to have to rework their algo?

[activeco]

Originally Posted by Tech Manager

I think we need to be careful to avoid flippant comments about proxies and other "origins" of traffic. Unless we are accurate we tend to promote the myth more than the reality.

Exactly. The same with blocking.
One thing for sure: The huge majority of attacks/spam don't use it's own IP-address.

[crankydave]

Originally Posted by janeth

And why should Google have to allow anyone on to their property that causes them to have to rework their algo?

First of all because businesses need to adapt their business models to meet the environment and not the other way around.

Secondly, because what John Mu suggested cannot be done. "Individuals" are not assigned unique IP's. You cannot block an "individual abuser". Blocking any IP blocks a region or geographic area.

Thirdly, it requires site owners to enable "theft" and/or potential "fraud" in order to be "compliant".

Finally, Google themselves block what content is delivered to paricular users in particular regions.

ETA... One final point, geotargeting is OK. Isn't this the same thing?

Dave

[Tech Manager]

Originally Posted by crankydave

First of all because businesses need to adapt their business models to meet the environment and not the other way around.

Secondly, because what John Mu suggested cannot be done. "Individuals" are not assigned unique IP's. You cannot block an "individual abuser". Blocking any IP blocks a region or geographic area.

Thirdly, it requires site owners to enable "theft" and/or potential "fraud" in order to be "compliant".

Finally, Google themselves block what content is delivered to paricular users in particular regions.

ETA... One final point, geotargeting is OK. Isn't this the same thing?

Dave

Outstanding rebuttal Dave. Excellent post. Completely on the mark.

If my business model is designed to protect my customers, networks, servers and websites then "Big Brother Search Engine" needs to build a better algorithm to reflect that model isntead of penalizing it.

The only way to block an individual is to place them in a prison cell and take their computer and communications away. When my servers are under attack I contemplate the SOB behind the attack, but I do so in my spare time. My immediate response is top stop the attack. That may require my blocking the soulless IP, IP Range or country.

The biggest need on the internet today is improved security. Google concerns itself with fraud preventioin when it comes to their revenue flows. We have to do the same thing. If Google's business model is designed around us lowering our security, being reactive instead of proactive, it is Google that needs to change.

I like Dave's last point the best: "...geotargeting is OK. Isn't this the same thing?" Yes, this is the same thing.

Google has a variety of locale specific search engines. A site can rank differently in the UK, Germany, USA, Mexico, etc., as it should.

Google may be the Big Boy on the block, but it is unacceptable to also have them as the Big Brother. They cannot rule the net. I'd love to see some of the major new networks take this subject on: "Google requires all websites to open themselves up to fraud, identity theft, hack attacks, and zombiism, in order to stay indexed in Google search results. News at 11."

[Tech Manager]

Originally Posted by activeco

Exactly. The same with blocking.
One thing for sure: The huge majority of attacks/spam don't use it's own IP-address.

That's true. The difference is, I actually know what I am talking about.

[janeth]

Originally Posted by crankydave

First of all because businesses need to adapt their business models to meet the environment and not the other way around.

Secondly, because what John Mu suggested cannot be done. "Individuals" are not assigned unique IP's. You cannot block an "individual abuser". Blocking any IP blocks a region or geographic area.

Thirdly, it requires site owners to enable "theft" and/or potential "fraud" in order to be "compliant".

Finally, Google themselves block what content is delivered to paricular users in particular regions.

ETA... One final point, geotargeting is OK. Isn't this the same thing?

Dave

I have nothing left to say. Guess you win, against me anyway. Google might be another story.

[nullvariable]

To quote Matt Cutts (A quick word about cloaking) :

"Cloaking is serving different content to users than to
search engines."

Maile Ohye states on The Official Google Webmaster blog (Official Google Webmaster Central Blog: How Google defines IP delivery, geolocation, and cloaking) :

"Geolocation: Serving targeted/different content to users based on
their location. As a webmaster, you may be able to determine a user's
location from preferences you've stored in their cookie, information
pertaining to their login, or their IP address. For example, if your
site is about baseball, you may use geolocation techniques to
highlight the Yankees to your users in New York.

The key is to treat Googlebot as you would a typical user from a similar location, IP range, etc. (i.e. don't treat Googlebot as if it
came from its own separate country—that's cloaking)."

I have a client for whom I wrote a simple script to make their contact page only throw an error to a specific country because of spam. This client does not have any interest in serving clients from this particular country and receives an extremely large amount of "we want to offer your business our services" type emails everyday. I believe that this fits in the Geolocation guideline, not the cloaking guideline since we do not make any distinction between a spider using those IP ranges or the Googlebot, individual users etc.

[activeco]

Originally Posted by Tech Manager

That's true. The difference is, I actually know what I am talking about.

Uh oh, getting personal. I am afraid your second sentence is not true.

What you don't see is that I support IP blocking and don't agree with the unfortunate post of JohnMu.
If a site blocks an IP range including googlebot, then it has nothing to do with cloaking and it is in line with Google's view:Official Google Webmaster Central Blog: How Google defines IP delivery, geolocation, and cloaking.
It is indeed Google's problem how to deal with results shown in locations where Google doesn't send bots from and the problem shouldn't be pushed to webmasters.

My disagreement with your post comes from posted prejudices about some countries being well known source of spam or viruses, which is absolutely impossible to prove.
Which of course, doesn't take away your legitimate right to block them.

[crankydave]

Originally Posted by janeth

I have nothing left to say. Guess you win, against me anyway. Google might be another story.

lol Janeth. It's not about winning to me. Your post was a great topic for discussion. Cudos!

This is just another instance (as I see it) whereby the means Google *may* wish to implement something that is simply wrong. I say *may* because the comment John Mu made didn't say *would* be considered cloaking.

A retail or ecommerce site that regularly receives attempts of fraudulent transactions from a particular IP range, attempts that cost them money, should have every right to block those attempts without having to fear being "punished" by a business model that cannot or will not adapt to it.

The ends do not justify the means. In this case, the ends being solely to benefit the Google business model.

Attempting to prevent illegal activities is by no stretch of the imagination "evil". Last I checked both theft and attempted fraud are illegal activities.

Dave

[Tech Manager]

Originally Posted by activeco

Uh oh, getting personal. I am afraid your second sentence is not true.

What you don't see is that I support IP blocking and don't agree with the unfortunate post of JohnMu.
If a site blocks an IP range including googlebot, then it has nothing to do with cloaking and it is in line with Google's view:Official Google Webmaster Central Blog: How Google defines IP delivery, geolocation, and cloaking.
It is indeed Google's problem how to deal with results shown in locations where Google doesn't send bots from and the problem shouldn't be pushed to webmasters.

My disagreement with your post comes from posted prejudices about some countries being well known source of spam or viruses, which is absolutely impossible to prove.
Which of course, doesn't take away your legitimate right to block them.

We'll just have to agree to disagree (an your second sentence is incorrect).

[nullvariable]

How to ban IP addresses WITHOUT banning GoogleBot - Crawling, indexing, and ranking | Google Groups

Hi everyone!
After a bit of double-checking, I have a clarification where I was
mistaken. Sorry about the confusion!

The important part is that you do not treat the Googlebots any
different than other users from that region. So if your site blocks
users in the region where the Googlebot comes from (based on the IP
address and your IP/Location lookups), you should be blocking it as
well. Blocking users outside of the Googlebot's region would generally
be ok.

For more questions, feel free to post here, thanks!

John

Its nice to be right sometimes

[Webnauts]

Originally Posted by nullvariable

How to ban IP addresses WITHOUT banning GoogleBot - Crawling, indexing, and ranking | Google Groups

Its nice to be right sometimes

Great job man!

By the way I forgot to mention in this thread that you are the Information Architect of my companies SEO Workers & Webnauts Net.

If anyone still has any questions, he is here in person, so you can ask him yourselves.

[AVC]

There are ways to block bad bots without blocking entire nations, so Google is correct if in fact they do punish webmasters blocking entire nations.

Why should they include you in the SERP's if your site does not present content to their users ?

This is simply common sense, not rocket science.

[nullvariable]

Originally Posted by AVC

There are ways to block bad bots without blocking entire nations, so Google is correct if in fact they do punish webmasters blocking entire nations.

Why should they include you in the SERP's if your site does not present content to their users ?

This is simply common sense, not rocket science.

exactly. if you for some reason want to block a whole country, go for it but don't try to whitelist the Googlebot of that country

[kgun]

Originally Posted by janeth

This is a bit much.

Google says they will ban you if you block people from viewing your site.

Google Says Blocking Countries Outside of the US is Against Policies

If you are being spammed by a bunch of guys in India and would like to block the whole country, you better rethink.

I have noted the following.

• While you are working on a site and block the outside world in .htaccess, your site is not indexed by GoogleBOT.
• That may even happen if you block enough IP regions like I do on ForumNorway that is no longer indexed by Google:
  
  site:forumnorway.com
  
  "Your search - site:forumnorway.com - did not match any documents". But that is my extranet, so it need not be indexed.
• Is there a limit on the number of IP regions blocked or

is it a new rumour, though from a fairly well informed site citing

"A Google Groups thread has a webmaster who has been receiving a lot of rogue spider attacks from the Africa region. He wants to go as far as ban the whole continent of Africa. But he is concerned that by doing so, he will also hurt his Google rankings".

My forum was indexed by Google when it was "open to the world" and I blocked a lot of unique IP's and IP regions.



To sum up:

1. Is there a limit on the number of Ip regions blocked or
2. is it a new rumor from a fairly credible soruce?

Since I am preparing for a holliday and in a hurry, I have had no time to read the other responses, so my post may be irrelevant.

[crankydave]

Originally Posted by AVC

There are ways to block bad bots without blocking entire nations, so Google is correct if in fact they do punish webmasters blocking entire nations.

Why should they include you in the SERP's if your site does not present content to their users ?

This is simply common sense, not rocket science.

But bad bots are not the sole or even neccessarily the primary reason for blocking IP's. Forums deny access to IP's and IP ranges all the time. When they do, they are not blocking the "individual" the are blocking an entire the region using that IP.

If they don't want to include a particular site in the SERP's for a particular area that's fine. However, not including the site in the SERP's does not mean someone cannot reach the site. Now if you're trying to suggest that Google shouldn't include a site in the SERP's at all anywhere, that's just plain wrong on so many levels.

What's simply common sense is site owners being proactive about deterring/preventing theft and fraud without having to fear being labeled as "noncompliant" and "punished" for it.

Dave

[activeco]

Originally Posted by kgun

I have noted the following.[LIST][*]While you are working on a site and block the outside world in .htaccess, you site is non indexed by GoogleBOT.[*]That may even happen if you block enough IP regions like I do on ForumNorway that is no longer indexed by Google:

site:forumnorway.com
...
My forum was indexed by Google when it was "open to the world" and I blocked a lot of unique IP's and IP regions.

You probably blocked Googlebot too. Check http://www.iplists.com/google.txt or even better find googlebot in your
earlier log files and release at least one region it comes most frequently from.

[kgun]

Thank you. On my todo list when I come from holiday

or thinking again, shall the posts on an extranet be index- and archivable?

[AVC]

Originally Posted by crankydave

But bad bots are not the sole or even necessarily the primary reason for blocking IP's. Forums deny access to IP's and IP ranges all the time. When they do, they are not blocking the "individual" the are blocking an entire the region using that IP.

If they don't want to include a particular site in the SERP's for a particular area that's fine. However, not including the site in the SERP's does not mean someone cannot reach the site. Now if you're trying to suggest that Google shouldn't include a site in the SERP's at all anywhere, that's just plain wrong on so many levels.

What's simply common sense is site owners being proactive about deterring/preventing theft and fraud without having to fear being labeled as "noncompliant" and "punished" for it.

Dave

I know of a forum that only makes content viewable to USA bikers, maybe North America only, I'm not sure, but Google should not show results worldwide for this site to their users.

Would you trust this showing in the results if you were running Google, is that serving your users if you were to show this in the results worldwide Dave ?

Yahoo and Google are now warning users of spyware infested sites that may damage their users computers and a lot of other stuff in the SERP's, so engines are advising users about sites now days.

Should Google just list sites blocking nations with the disclaimer that "you may not be able to view this site, use a proxy" ????

[activeco]

Originally Posted by kgun

... shall the posts on an extranet be index- and archivable?

OK I see the connection with the thread.

Extranets are usually closed behind firewalls so there is no access from general public as well as from outside bots.
However there is a thing called Google Search Appliance that provides Google technology for closed environments.
With it, an organization gets completely internal Google search engine.

[kgun]

Originally Posted by crankydave

Except that "users" are not assigned individual IP's. Blocking a particular IP or IP range prevents everyone in the "region" from access not just the individual.

Dave

And blocking an unique IP may block out a lot of sites if they are hosted on a shared server.

[kgun]

Originally Posted by Webnauts

Already my team Information Architect is working on a script that blocks suspicious bots by IP for 24 hours, and then get unblocked. This thread is a good hint to extend that also for bad visitors. I will discuss that with him tomorrow to see what and how that would be possible.

I tried that manually without any success.

If you use a script, may be the best is to write a spider trap: Scripts: Spider Blocking :: .htaccess, PHP, Block Bad Bots with .htaccess or PHP

[nullvariable]

Originally Posted by kgun

I tried that manually without any success.

If you use a script, may be the best is to write a spider trap: Scripts: Spider Blocking :: .htaccess, PHP, Block Bad Bots with .htaccess or PHP

Actually the script that I am writing will be directing bad requests and IPs to a Spam poison/spider trap part of the server. I discovered that someone wrote a really crappy crawler and was crawling my site with it. They were following links with ":" and it didn't even correctly remove the " at the end of some requests. Its pretty easy to setup a robots.txt and then drop in a link that can be spidered by bad bots. My script just takes hits to this link and blocks them and forwards them off to the spam traps/honey pots. It lowers the load on the main site and fills spammers databases with junk. But Googlebot and others that respect robots.txt will never run into all of this. Of course there are the actions of individual users to consider and one way I am thinking of doing it is to have all the error pages log hits and if someone hits an error page more than X times they get blocked for 24 hours.

Most of the IPs that I see doing this are in China, India and other locations, where I really could care less about blocking some guy and his neighbors for a few hours. The likelyhood that they are actually surfing my site is pretty slim. Basically I will be looking to filter unusual (or typical bad bot) behaviours and then simple block the IP for 24 hours and forward it off to a poison trap or honey pot.

[kgun]

Now I invite all of you that have posted in this thread to rethink the issue. Most of us start out by:

allow from all IP regions.

Why not turn that around and

deny from all

and gradually open up IP regions.