iEntry 10th Anniversary Forum Rules Search
WebProWorld
Register FAQ Calendar Mark Forums Read
Google Discussion Forum Google Discussion forum is for topics specifically related to Google. There is a subforum dedicated to AdSense/AdWords subjects.

Share Thread: & Tags

Share Thread:

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 07-02-2008, 04:19 PM
janeth's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Jul 2003
Location: Colombia S.A
Posts: 5,746
janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4
Default Google Says Blocking Countries Outside of the US is Against

This is a bit much.

Google says they will ban you if you block people from viewing your site.

Google Says Blocking Countries Outside of the US is Against Policies

If you are being spammed by a bunch of guys in India and would like to block the whole country, you better rethink.
__________________
Custom Web Design | CRM | Wristbands
Reply With Quote
  #2 (permalink)  
Old 07-02-2008, 05:14 PM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

I suppose Google should stop helping China block content from outside the country.

I do a lot of work in regards to blocking traffic by country. So far I have seen absolutely no effect on any website taking appropriate steps to block an IP, an IP range or an entire country.
__________________
I use Country IP Blocks as added security for my networks and servers.
Reply With Quote
  #3 (permalink)  
Old 07-02-2008, 05:20 PM
janeth's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Jul 2003
Location: Colombia S.A
Posts: 5,746
janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Tech Manager View Post
I suppose Google should stop helping China block content from outside the country.

I do a lot of work in regards to blocking traffic by country. So far I have seen absolutely no effect on any website taking appropriate steps to block an IP, an IP range or an entire country.
I agree and have had to do it for clients in the past. Sometimes it is the only way to save a site.
__________________
Custom Web Design | CRM | Wristbands
Reply With Quote
  #4 (permalink)  
Old 07-02-2008, 05:32 PM
crankydave's Avatar
Moderator
WebProWorld Moderator
 
Join Date: Aug 2004
Location: Playing with fire!
Posts: 4,120
crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6
Default Re: Google Says Blocking Countries Outside of the US is Against

Let's see... an ecommerce site that only services a particular region, regularly receives attempts of credit card fraud from outside that region, blocks those particular IP ranges, is going to be considered to be cloaking?

Is it April 1st or sheer lunacy?

Dave
Reply With Quote
  #5 (permalink)  
Old 07-02-2008, 05:46 PM
wige's Avatar
Moderator
WebProWorld Moderator
 
Join Date: Jun 2006
Location: United States
Posts: 2,376
wige RepRank 5wige RepRank 5wige RepRank 5wige RepRank 5wige RepRank 5wige RepRank 5
Default Re: Google Says Blocking Countries Outside of the US is Against

But think about this from Google's standpoint. As a search engine, Google can't tell what parts of the world you are blocking from accessing your site, at least reliably, and Google may not have a system in place to filter out your site from the results when people in those parts of the world run searches.

Google has stated repeatedly that their goal is to provide the best possible search results, and avoid users from clicking into an error message. If IP filtering is perceived by Google to make their results look faulty to those who are blocked, would you not expect them to take action, up to dropping the sites that do this?
__________________
The best way to learn anything, is to question everything.
Interestingly Average Security Blog
Reply With Quote
  #6 (permalink)  
Old 07-02-2008, 05:50 PM
janeth's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Jul 2003
Location: Colombia S.A
Posts: 5,746
janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by wige View Post
But think about this from Google's standpoint. As a search engine, Google can't tell what parts of the world you are blocking from accessing your site, at least reliably, and Google may not have a system in place to filter out your site from the results when people in those parts of the world run searches.

Google has stated repeatedly that their goal is to provide the best possible search results, and avoid users from clicking into an error message. If IP filtering is perceived by Google to make their results look faulty to those who are blocked, would you not expect them to take action, up to dropping the sites that do this?
It's what I was thinking as well, but for the site owner it's a loose, loose situation.
__________________
Custom Web Design | CRM | Wristbands
Reply With Quote
  #7 (permalink)  
Old 07-02-2008, 05:59 PM
crankydave's Avatar
Moderator
WebProWorld Moderator
 
Join Date: Aug 2004
Location: Playing with fire!
Posts: 4,120
crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by wige View Post
But think about this from Google's standpoint. As a search engine, Google can't tell what parts of the world you are blocking from accessing your site, at least reliably, and Google may not have a system in place to filter out your site from the results when people in those parts of the world run searches.

Google has stated repeatedly that their goal is to provide the best possible search results, and avoid users from clicking into an error message. If IP filtering is perceived by Google to make their results look faulty to those who are blocked, would you not expect them to take action, up to dropping the sites that do this?
I see, Google's business model can't "reliably" do something so the answer is to make it a violation of their guidelines to do so? That's rich.

Quote:
Dear Google

These people keep coming to "my house" and stealing from me. Is it okay if keep them out of my house?

Dave
Quote:
Dear Dave

No it is not. That would be evil. You have to let them in your house or we will punish you.

Google
Reply With Quote
  #8 (permalink)  
Old 07-02-2008, 06:27 PM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

I'm beginning to wonder whether George Orwell wrote the rulebook for Google.

Google might want to be the best, most accurate Search Engine around, but they aren't going to stay their if they require people to dumbdown their security to accomodate Google results. If Google wants to penalize me for blocking Russia, China and Korea they can do so without penalizing websites with the rest of the globe.
__________________
I use Country IP Blocks as added security for my networks and servers.
Reply With Quote
  #9 (permalink)  
Old 07-02-2008, 06:33 PM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

I haven't noticed anything in Googles rulebook that declares cloaking to include denying access to certain IP ranges.

Cloaking, sneaky Javascript redirects, and doorway pages

I suppose you could stretch the rule to suggest serving no content is the same as serving different content.
__________________
I use Country IP Blocks as added security for my networks and servers.
Reply With Quote
  #10 (permalink)  
Old 07-02-2008, 06:37 PM
Webnauts's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Aug 2003
Location: Worldwide
Posts: 7,968
Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

John Mu at Google said:

"Blocking all users outside of the US from being able to see your site would likely be considered cloaking and would be against our Webmaster Guidelines. Instead of blocking these users automatically, I would recommend that you add blocks based on the user's activity, not based on his location. "

And I think that is pretty fair.
Reply With Quote
  #11 (permalink)  
Old 07-02-2008, 06:50 PM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Webnauts View Post
John Mu at Google said:

"Blocking all users outside of the US from being able to see your site would likely be considered cloaking and would be against our Webmaster Guidelines. Instead of blocking these users automatically, I would recommend that you add blocks based on the user's activity, not based on his location. "

And I think that is pretty fair.
I have no sites that only allow USA access, but regardless, I think John Mu is entirely incorrect. Sure, blocking based on user activity might be preferable, but it can be impossible for some webmasters. I manage a significant number of servers and networks. Malicious traffic from Russia, China, North Korea and a myriad of other locales can be highly disruptive to email servers, websites, intranets, etc. Is it more expedient for me to spend my time actively responding to the behavior of traffic that is 99.9% malicious or is my time better spent blocking the traffic entirely?

As an IT Manager I don't plan security and protection of sensitive data around the whims of Google, Yahoo, MSN or others. I have an obligation to the clients I support to keep their sites, servers and traffic as safe as possible.

Strong Security is multifaceted. When it calls for blocking a range of IPs or an entire country I will do so.

Is Google going to penalize you for relying on SBL, PBL, XBL lists to identify IP addresses and IP ranges of spammers? Are they going to penalize you for blocking known IP ranges of comment spammers, brute force attackers and email harvesters?

Prevention is the first line of defense. If my security becomes solely based on behavior instead of proactive prevention, I have failed.
__________________
I use Country IP Blocks as added security for my networks and servers.

Last edited by Tech Manager; 07-02-2008 at 06:58 PM. Reason: typo
Reply With Quote
  #12 (permalink)  
Old 07-02-2008, 06:58 PM
crankydave's Avatar
Moderator
WebProWorld Moderator
 
Join Date: Aug 2004
Location: Playing with fire!
Posts: 4,120
crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Webnauts View Post
John Mu at Google said:

"Blocking all users outside of the US from being able to see your site would likely be considered cloaking and would be against our Webmaster Guidelines. Instead of blocking these users automatically, I would recommend that you add blocks based on the user's activity, not based on his location. "

And I think that is pretty fair.
Except that "users" are not assigned individual IP's. Blocking a particular IP or IP range prevents everyone in the "region" from access not just the individual.

Dave
Reply With Quote
  #13 (permalink)  
Old 07-02-2008, 06:59 PM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

Excellent point Dave.
__________________
I use Country IP Blocks as added security for my networks and servers.
Reply With Quote
  #14 (permalink)  
Old 07-02-2008, 07:33 PM
WebProWorld Veteran
 
Join Date: Oct 2005
Location: England
Posts: 846
Martinscholes RepRank 2
Default Re: Google Says Blocking Countries Outside of the US is Against

If someone in -say- California attacks a website in Washington State but uses spoof ips of -say- Holland, then the website in Washington State would block Dutch ips, whereas the real villain is in California, laughing his socks off.
Reply With Quote
  #15 (permalink)  
Old 07-02-2008, 07:39 PM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

A spoofed IP attack is not quite as simple as your post would lead someone to believe. IP spoofing doesn't truly hide your actual IP. Generally speaking, IP spoofing does not generally allow for a normal network connection. This type of activity generally entails a blind spoof.

While IP spoofing can happen it is more common to see various other attacks.
__________________
I use Country IP Blocks as added security for my networks and servers.
Reply With Quote
  #16 (permalink)  
Old 07-02-2008, 07:49 PM
Webnauts's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Aug 2003
Location: Worldwide
Posts: 7,968
Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by crankydave View Post
Except that "users" are not assigned individual IP's. Blocking a particular IP or IP range prevents everyone in the "region" from access not just the individual.

Dave
Already my team Information Architect is working on a script that blocks suspicious bots by IP for 24 hours, and then get unblocked. This thread is a good hint to extend that also for bad visitors. I will discuss that with him tomorrow to see what and how that would be possible.
Reply With Quote
  #17 (permalink)  
Old 07-02-2008, 07:50 PM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Webnauts View Post
Already my team Information Architect is working on a script that blocks suspicious bots by IP for 24 hours, and then get unblocked. This thread is a good hint to extend that also for bad visitors. I will discuss that with him tomorrow to see what and how that would be possible.
Let us know.
__________________
I use Country IP Blocks as added security for my networks and servers.
Reply With Quote
  #18 (permalink)  
Old 07-02-2008, 08:06 PM
Webnauts's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Aug 2003
Location: Worldwide
Posts: 7,968
Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

I was also thinking about something else.

We get in average 8-10 emails a day from Indian companies, offering links building, etc services, mentioning that they conform to the US CAN-SPAM law. We report all those mails as spam, since that law is not valid in Germany were we are located. In Germany unsolicited emails, no matter which nature, are strictly forbidden.

Now they began using our contact form. We created an IP based script which we wanted to add in our contact form, which would redirect all India visitors to our homepage instead.

Before we had the chance to go live, we read the thread at Googles Webmaster Help Center, which Janeth shared here too.

Looks like we only have two options:

1. Disallow our contact page in our robots.txt and implement our script.

or

2. Extend the script by blocking the transaction of the mail if the visitor comes from India.

In both cases it will not be cloaking. I assume the 2nd option would be possible and I obviously prefer, since I do not want my contact page to be de-indexed. I will check that with our IA too.

Last edited by Webnauts; 07-02-2008 at 08:09 PM.
Reply With Quote
  #19 (permalink)  
Old 07-02-2008, 08:13 PM
Webnauts's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Aug 2003
Location: Worldwide
Posts: 7,968
Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Martinscholes View Post
If someone in -say- California attacks a website in Washington State but uses spoof ips of -say- Holland, then the website in Washington State would block Dutch ips, whereas the real villain is in California, laughing his socks off.
This is what we also planning to implement: Http:BL Application Programming Interface (API) | Project Honey Pot
Worth to have a look.

Upon the chance, allow me to refer you all to the excellent service:
Distributed Spam Harvester Tracking Network | Project Honey Pot
Reply With Quote
  #20 (permalink)  
Old 07-02-2008, 08:19 PM
WebProWorld Veteran
 
Join Date: Jul 2004
Posts: 981
activeco RepRank 2
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Tech Manager View Post
IP spoofing doesn't truly hide your actual IP.
But a good proxy does and many backdoors are used exactly for that.
Martinscholes is right, most of the villain IP's are being intentionally chosen from another side of the globe.
__________________
Impossible? You just underestimate the time.
Reply With Quote
  #21 (permalink)  
Old 07-02-2008, 08:24 PM
Webnauts's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Aug 2003
Location: Worldwide
Posts: 7,968
Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by activeco View Post
But a good proxy does and many backdoors are used exactly for that.
Martinscholes is right, most of the villain IP's are being intentionally chosen from another side of the globe.
This solution is not the best ever, but it still helps to block a respective number of proxy servers:

RewriteCond %{HTTP:VIA} !^$ [OR]
RewriteCond %{HTTP:FORWARDED} !^$ [OR]
RewriteCond %{HTTP:USERAGENT_VIA} !^$ [OR]
RewriteCond %{HTTP:X_FORWARDED_FOR} !^$ [OR]
RewriteCond %{HTTP:PROXY_CONNECTION} !^$ [OR]
RewriteCond %{HTTP:XPROXY_CONNECTION} !^$ [OR]
RewriteCond %{HTTP:HTTP_PC_REMOTE_ADDR} !^$ [OR]
RewriteCond %{HTTP:HTTP_CLIENT_IP} !^$
RewriteRule ^(.*)$ - [F]
Reply With Quote
  #22 (permalink)  
Old 07-02-2008, 08:28 PM
WebProWorld Veteran
 
Join Date: Jul 2004
Posts: 981
activeco RepRank 2
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Webnauts View Post
This solution is not the best ever, but it still helps to block a respective number of proxy servers:
No 'professional' proxy reveals itself in http headers.
__________________
Impossible? You just underestimate the time.
Reply With Quote
  #23 (permalink)  
Old 07-02-2008, 09:00 PM
Feydakin's Avatar
WebProWorld Veteran
WebProWorld MVP
 
Join Date: Jan 2005
Location: ho jeez!!
Posts: 871
Feydakin RepRank 5Feydakin RepRank 5Feydakin RepRank 5Feydakin RepRank 5Feydakin RepRank 5Feydakin RepRank 5
Default Re: Google Says Blocking Countries Outside of the US is Against

Google India is more than welcome to drop my site from it's serps.. I don't sell to India.. Or Korea, or the Ukraine.. So all of those Google.s are more than welcome to drop me from their index.. It would save us a LOT of hassle on the customer service end answering emails trying to explain "why" we refuse to ship there..
__________________
Steve : Animal Charms Animal Jewelry | Internet Marketing
I'm smelling a whole lot of if coming off of this plan.
Reply With Quote
  #24 (permalink)  
Old 07-02-2008, 09:23 PM
crankydave's Avatar
Moderator
WebProWorld Moderator
 
Join Date: Aug 2004
Location: Playing with fire!
Posts: 4,120
crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6
Default Re: Google Says Blocking Countries Outside of the US is Against

hmmmmmmmmmmm...

Why should I or any other website owner have to allow anyone or anything access to their "property" to be "compliant" with a set of "rules' or "guidelines" SPECIFICALLY designed to promote their own interests ESPECIALLY when they're stealing or trying to commit fraud?

Tell you what John... name a single IP you've blocked/redirected/anything that targets ONLY the individual "abusing" it and not everyone in that region on the same IP.

Dave
Reply With Quote
  #25 (permalink)  
Old 07-03-2008, 08:09 AM
Webnauts's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Aug 2003
Location: Worldwide
Posts: 7,968
Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by crankydave View Post
Tell you what John... name a single IP you've blocked/redirected/anything that targets ONLY the individual "abusing" it and not everyone in that region on the same IP.
I fully agree with that Dave. As you know I am an advocate of Web Content Accessibility, and if I would do otherwise as you said, I will be violating those standards.
Reply With Quote
  #26 (permalink)  
Old 07-03-2008, 09:18 AM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by activeco View Post
But a good proxy does and many backdoors are used exactly for that.
Martinscholes is right, most of the villain IP's are being intentionally chosen from another side of the globe.
MartinScholes is semi-correct. Proxies are an all together issue and can be handled differently. Proxies exist all over the world. There are plenty of them right here in the United States. Many non-IT/network people tend to regurgitateless than accurate information and sometimes downright myths and urban legends about security.

Malcious traffic can be filtered in a multitude of ways with a wide variety of rulesets. Proxy detection filters can be added to weed out known and unknown proxies.

While proxies still pose a significant threat they are manageable.

Proxies make up a comparatively smaller portion of malicious/spam/hacking traffic than you might think. Currently, a much bigger problem is malcious Botnets. Botnets create a much more randomized assault as they rely on infected machines to spew their traffic.

I keep hearing the proxy argument position as why country or IP range blocks simply don't work. It's an internet urban legend.

The argument goes something like this, "If you block an IP range a smart hacker will just use a proxy to get around your protection; therefore it is better not do this."

This is circular reasononing. It is much like saying, "Don't bother putting a lock on your front door because an intruder will then use your backdoor."

Ok, so I'll put a lock on my backdoor too!

"No, don't bother putting a lock on your backdoor because an intruder will break in through one of your windows."

Well then, he's going to set off my silent alarm, alert my doberman and if he survives the dog he's going to be staring down a loaded gun. Problem solved.

As I've said many times, security is multifaceted. It starts with the concept of protection and permissions. Whom do you want to allow into your home, website, servers and networks? It then incorporates several layers of protection: hardware and software firewalls, blacklists and white lists (which can include network ranges and/or countries), proxy detection, XBL, PBL, XBL lists, traffic behavior patterns and response, secure coding, properly validated variables, bad-behavior-redirection, and myriad other proactive and reactive layers.

I think we need to be careful to avoid flippant comments about proxies and other "origins" of traffic. Unless we are accurate we tend to promote the myth more than the reality.
__________________
I use Country IP Blocks as added security for my networks and servers.
Reply With Quote
  #27 (permalink)  
Old 07-03-2008, 09:46 AM
janeth's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Jul 2003
Location: Colombia S.A
Posts: 5,746
janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by crankydave View Post
hmmmmmmmmmmm...

Why should I or any other website owner have to allow anyone or anything access to their "property" to be "compliant" with a set of "rules' or "guidelines" SPECIFICALLY designed to promote their own interests ESPECIALLY when they're stealing or trying to commit fraud?
And why should Google have to allow anyone on to their property that causes them to have to rework their algo?
__________________
Custom Web Design | CRM | Wristbands
Reply With Quote
  #28 (permalink)  
Old 07-03-2008, 09:49 AM
WebProWorld Veteran
 
Join Date: Jul 2004
Posts: 981
activeco RepRank 2
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Tech Manager View Post
I think we need to be careful to avoid flippant comments about proxies and other "origins" of traffic. Unless we are accurate we tend to promote the myth more than the reality.
Exactly. The same with blocking.
One thing for sure: The huge majority of attacks/spam don't use it's own IP-address.
__________________
Impossible? You just underestimate the time.
Reply With Quote
  #29 (permalink)  
Old 07-03-2008, 10:04 AM
crankydave's Avatar
Moderator
WebProWorld Moderator
 
Join Date: Aug 2004
Location: Playing with fire!
Posts: 4,120
crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by janeth View Post
And why should Google have to allow anyone on to their property that causes them to have to rework their algo?
First of all because businesses need to adapt their business models to meet the environment and not the other way around.

Secondly, because what John Mu suggested cannot be done. "Individuals" are not assigned unique IP's. You cannot block an "individual abuser". Blocking any IP blocks a region or geographic area.

Thirdly, it requires site owners to enable "theft" and/or potential "fraud" in order to be "compliant".

Finally, Google themselves block what content is delivered to paricular users in particular regions.

ETA... One final point, geotargeting is OK. Isn't this the same thing?

Dave

Last edited by crankydave; 07-03-2008 at 10:31 AM.
Reply With Quote
  #30 (permalink)  
Old 07-03-2008, 10:57 AM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by crankydave View Post
First of all because businesses need to adapt their business models to meet the environment and not the other way around.

Secondly, because what John Mu suggested cannot be done. "Individuals" are not assigned unique IP's. You cannot block an "individual abuser". Blocking any IP blocks a region or geographic area.

Thirdly, it requires site owners to enable "theft" and/or potential "fraud" in order to be "compliant".

Finally, Google themselves block what content is delivered to paricular users in particular regions.

ETA... One final point, geotargeting is OK. Isn't this the same thing?

Dave

Outstanding rebuttal Dave. Excellent post. Completely on the mark.

If my business model is designed to protect my customers, networks, servers and websites then "Big Brother Search Engine" needs to build a better algorithm to reflect that model isntead of penalizing it.

The only way to block an individual is to place them in a prison cell and take their computer and communications away. When my servers are under attack I contemplate the SOB behind the attack, but I do so in my spare time. My immediate response is top stop the attack. That may require my blocking the soulless IP, IP Range or country.

The biggest need on the internet today is improved security. Google concerns itself with fraud preventioin when it comes to their revenue flows. We have to do the same thing. If Google's business model is designed around us lowering our security, being reactive instead of proactive, it is Google that needs to change.

I like Dave's last point the best: "...geotargeting is OK. Isn't this the same thing?" Yes, this is the same thing.

Google has a variety of locale specific search engines. A site can rank differently in the UK, Germany, USA, Mexico, etc., as it should.

Google may be the Big Boy on the block, but it is unacceptable to also have them as the Big Brother. They cannot rule the net. I'd love to see some of the major new networks take this subject on: "Google requires all websites to open themselves up to fraud, identity theft, hack attacks, and zombiism, in order to stay indexed in Google search results. News at 11."
__________________
I use Country IP Blocks as added security for my networks and servers.
Reply With Quote
  #31 (permalink)  
Old 07-03-2008, 10:59 AM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by activeco View Post
Exactly. The same with blocking.
One thing for sure: The huge majority of attacks/spam don't use it's own IP-address.
That's true. The difference is, I actually know what I am talking about.
__________________
I use Country IP Blocks as added security for my networks and servers.
Reply With Quote
  #32 (permalink)  
Old 07-03-2008, 11:02 AM
janeth's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Jul 2003
Location: Colombia S.A
Posts: 5,746
janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4janeth RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by crankydave View Post
First of all because businesses need to adapt their business models to meet the environment and not the other way around.

Secondly, because what John Mu suggested cannot be done. "Individuals" are not assigned unique IP's. You cannot block an "individual abuser". Blocking any IP blocks a region or geographic area.

Thirdly, it requires site owners to enable "theft" and/or potential "fraud" in order to be "compliant".

Finally, Google themselves block what content is delivered to paricular users in particular regions.

ETA... One final point, geotargeting is OK. Isn't this the same thing?

Dave
I have nothing left to say. Guess you win, against me anyway. Google might be another story.
__________________
Custom Web Design | CRM | Wristbands
Reply With Quote
  #33 (permalink)  
Old 07-03-2008, 11:33 AM
nullvariable's Avatar
WebProWorld Pro
 
Join Date: Sep 2005
Location: Greenville, SC
Posts: 216
nullvariable RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

To quote Matt Cutts (A quick word about cloaking) :
Quote:
"Cloaking is serving different content to users than to
search engines."
Maile Ohye states on The Official Google Webmaster blog (Official Google Webmaster Central Blog: How Google defines IP delivery, geolocation, and cloaking) :
Quote:
"Geolocation: Serving targeted/different content to users based on
their location. As a webmaster, you may be able to determine a user's
location from preferences you've stored in their cookie, information
pertaining to their login, or their IP address. For example, if your
site is about baseball, you may use geolocation techniques to
highlight the Yankees to your users in New York.

The key is to treat Googlebot as you would a typical user from a similar location, IP range, etc. (i.e. don't treat Googlebot as if it
came from its own separate country—that's cloaking)."
I have a client for whom I wrote a simple script to make their contact page only throw an error to a specific country because of spam. This client does not have any interest in serving clients from this particular country and receives an extremely large amount of "we want to offer your business our services" type emails everyday. I believe that this fits in the Geolocation guideline, not the cloaking guideline since we do not make any distinction between a spider using those IP ranges or the Googlebot, individual users etc.
__________________
Blog | Web Consulting
Reply With Quote
  #34 (permalink)  
Old 07-03-2008, 11:38 AM
WebProWorld Veteran
 
Join Date: Jul 2004
Posts: 981
activeco RepRank 2
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Tech Manager View Post
That's true. The difference is, I actually know what I am talking about.
Uh oh, getting personal. I am afraid your second sentence is not true.

What you don't see is that I support IP blocking and don't agree with the unfortunate post of JohnMu.
If a site blocks an IP range including googlebot, then it has nothing to do with cloaking and it is in line with Google's view:Official Google Webmaster Central Blog: How Google defines IP delivery, geolocation, and cloaking.
It is indeed Google's problem how to deal with results shown in locations where Google doesn't send bots from and the problem shouldn't be pushed to webmasters.

My disagreement with your post comes from posted prejudices about some countries being well known source of spam or viruses, which is absolutely impossible to prove.
Which of course, doesn't take away your legitimate right to block them.
__________________
Impossible? You just underestimate the time.

Last edited by activeco; 07-03-2008 at 11:41 AM.
Reply With Quote
  #35 (permalink)  
Old 07-03-2008, 11:59 AM
crankydave's Avatar
Moderator
WebProWorld Moderator
 
Join Date: Aug 2004
Location: Playing with fire!
Posts: 4,120
crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by janeth View Post
I have nothing left to say. Guess you win, against me anyway. Google might be another story.
lol Janeth. It's not about winning to me. Your post was a great topic for discussion. Cudos!

This is just another instance (as I see it) whereby the means Google *may* wish to implement something that is simply wrong. I say *may* because the comment John Mu made didn't say *would* be considered cloaking.

A retail or ecommerce site that regularly receives attempts of fraudulent transactions from a particular IP range, attempts that cost them money, should have every right to block those attempts without having to fear being "punished" by a business model that cannot or will not adapt to it.

The ends do not justify the means. In this case, the ends being solely to benefit the Google business model.

Attempting to prevent illegal activities is by no stretch of the imagination "evil". Last I checked both theft and attempted fraud are illegal activities.

Dave
Reply With Quote
  #36 (permalink)  
Old 07-03-2008, 12:10 PM
WebProWorld Veteran
 
Join Date: Jan 2008
Posts: 304
Tech Manager RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by activeco View Post
Uh oh, getting personal. I am afraid your second sentence is not true.

What you don't see is that I support IP blocking and don't agree with the unfortunate post of JohnMu.
If a site blocks an IP range including googlebot, then it has nothing to do with cloaking and it is in line with Google's view:Official Google Webmaster Central Blog: How Google defines IP delivery, geolocation, and cloaking.
It is indeed Google's problem how to deal with results shown in locations where Google doesn't send bots from and the problem shouldn't be pushed to webmasters.

My disagreement with your post comes from posted prejudices about some countries being well known source of spam or viruses, which is absolutely impossible to prove.
Which of course, doesn't take away your legitimate right to block them.
We'll just have to agree to disagree (an your second sentence is incorrect).
__________________
I use Country IP Blocks as added security for my networks and servers.

Last edited by Tech Manager; 07-03-2008 at 12:22 PM. Reason: typo and tone
Reply With Quote
  #37 (permalink)  
Old 07-03-2008, 12:44 PM
nullvariable's Avatar
WebProWorld Pro
 
Join Date: Sep 2005
Location: Greenville, SC
Posts: 216
nullvariable RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

How to ban IP addresses WITHOUT banning GoogleBot - Crawling, indexing, and ranking | Google Groups
Quote:
Hi everyone!
After a bit of double-checking, I have a clarification where I was
mistaken. Sorry about the confusion!

The important part is that you do not treat the Googlebots any
different than other users from that region. So if your site blocks
users in the region where the Googlebot comes from (based on the IP
address and your IP/Location lookups), you should be blocking it as
well. Blocking users outside of the Googlebot's region would generally
be ok.

For more questions, feel free to post here, thanks!

John
Its nice to be right sometimes
__________________
Blog | Web Consulting
Reply With Quote
  #38 (permalink)  
Old 07-03-2008, 12:48 PM
Webnauts's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Aug 2003
Location: Worldwide
Posts: 7,968
Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4Webnauts RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by nullvariable View Post
Great job man!

By the way I forgot to mention in this thread that you are the Information Architect of my companies SEO Workers & Webnauts Net.

If anyone still has any questions, he is here in person, so you can ask him yourselves.
Reply With Quote
  #39 (permalink)  
Old 07-03-2008, 12:50 PM
AVC AVC is offline
WebProWorld Veteran
 
Join Date: Aug 2003
Location: USA
Posts: 488
AVC RepRank 0
Default Re: Google Says Blocking Countries Outside of the US is Against

There are ways to block bad bots without blocking entire nations, so Google is correct if in fact they do punish webmasters blocking entire nations.

Why should they include you in the SERP's if your site does not present content to their users ?

This is simply common sense, not rocket science.
Reply With Quote
  #40 (permalink)  
Old 07-03-2008, 12:53 PM
nullvariable's Avatar
WebProWorld Pro
 
Join Date: Sep 2005
Location: Greenville, SC
Posts: 216
nullvariable RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by AVC View Post
There are ways to block bad bots without blocking entire nations, so Google is correct if in fact they do punish webmasters blocking entire nations.

Why should they include you in the SERP's if your site does not present content to their users ?

This is simply common sense, not rocket science.
exactly. if you for some reason want to block a whole country, go for it but don't try to whitelist the Googlebot of that country
__________________
Blog | Web Consulting
Reply With Quote
  #41 (permalink)  
Old 07-03-2008, 01:00 PM
kgun's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: May 2005
Location: Norway
Posts: 6,635
kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by janeth View Post
This is a bit much.

Google says they will ban you if you block people from viewing your site.

Google Says Blocking Countries Outside of the US is Against Policies

If you are being spammed by a bunch of guys in India and would like to block the whole country, you better rethink.

I have noted the following.
  • While you are working on a site and block the outside world in .htaccess, your site is not indexed by GoogleBOT.
  • That may even happen if you block enough IP regions like I do on ForumNorway that is no longer indexed by Google:

    site:forumnorway.com

    "Your search - site:forumnorway.com - did not match any documents". But that is my extranet, so it need not be indexed.
  • Is there a limit on the number of IP regions blocked or
is it a new rumour, though from a fairly well informed site citing

"A Google Groups thread has a webmaster who has been receiving a lot of rogue spider attacks from the Africa region. He wants to go as far as ban the whole continent of Africa. But he is concerned that by doing so, he will also hurt his Google rankings".

My forum was indexed by Google when it was "open to the world" and I blocked a lot of unique IP's and IP regions.



To sum up:
  1. Is there a limit on the number of Ip regions blocked or
  2. is it a new rumor from a fairly credible soruce?
Since I am preparing for a holliday and in a hurry, I have had no time to read the other responses, so my post may be irrelevant.

Last edited by kgun; 07-03-2008 at 01:43 PM. Reason: Spelling
Reply With Quote
  #42 (permalink)  
Old 07-03-2008, 01:18 PM
crankydave's Avatar
Moderator
WebProWorld Moderator
 
Join Date: Aug 2004
Location: Playing with fire!
Posts: 4,120
crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6crankydave RepRank 6
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by AVC View Post
There are ways to block bad bots without blocking entire nations, so Google is correct if in fact they do punish webmasters blocking entire nations.

Why should they include you in the SERP's if your site does not present content to their users ?

This is simply common sense, not rocket science.
But bad bots are not the sole or even neccessarily the primary reason for blocking IP's. Forums deny access to IP's and IP ranges all the time. When they do, they are not blocking the "individual" the are blocking an entire the region using that IP.

If they don't want to include a particular site in the SERP's for a particular area that's fine. However, not including the site in the SERP's does not mean someone cannot reach the site. Now if you're trying to suggest that Google shouldn't include a site in the SERP's at all anywhere, that's just plain wrong on so many levels.

What's simply common sense is site owners being proactive about deterring/preventing theft and fraud without having to fear being labeled as "noncompliant" and "punished" for it.

Dave
Reply With Quote
  #43 (permalink)  
Old 07-03-2008, 01:25 PM
WebProWorld Veteran
 
Join Date: Jul 2004
Posts: 981
activeco RepRank 2
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by kgun View Post
I have noted the following.[LIST][*]While you are working on a site and block the outside world in .htaccess, you site is non indexed by GoogleBOT.[*]That may even happen if you block enough IP regions like I do on ForumNorway that is no longer indexed by Google:

site:forumnorway.com
...
My forum was indexed by Google when it was "open to the world" and I blocked a lot of unique IP's and IP regions.
You probably blocked Googlebot too. Check http://www.iplists.com/google.txt or even better find googlebot in your
earlier log files and release at least one region it comes most frequently from.
__________________
Impossible? You just underestimate the time.
Reply With Quote
  #44 (permalink)  
Old 07-03-2008, 01:42 PM
kgun's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: May 2005
Location: Norway
Posts: 6,635
kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Thank you. On my todo list when I come from holiday

or thinking again, shall the posts on an extranet be index- and archivable?

Last edited by kgun; 07-03-2008 at 01:49 PM.
Reply With Quote
  #45 (permalink)  
Old 07-03-2008, 01:48 PM
AVC AVC is offline
WebProWorld Veteran
 
Join Date: Aug 2003
Location: USA
Posts: 488
AVC RepRank 0
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by crankydave View Post
But bad bots are not the sole or even necessarily the primary reason for blocking IP's. Forums deny access to IP's and IP ranges all the time. When they do, they are not blocking the "individual" the are blocking an entire the region using that IP.

If they don't want to include a particular site in the SERP's for a particular area that's fine. However, not including the site in the SERP's does not mean someone cannot reach the site. Now if you're trying to suggest that Google shouldn't include a site in the SERP's at all anywhere, that's just plain wrong on so many levels.

What's simply common sense is site owners being proactive about deterring/preventing theft and fraud without having to fear being labeled as "noncompliant" and "punished" for it.

Dave
I know of a forum that only makes content viewable to USA bikers, maybe North America only, I'm not sure, but Google should not show results worldwide for this site to their users.

Would you trust this showing in the results if you were running Google, is that serving your users if you were to show this in the results worldwide Dave ?

Yahoo and Google are now warning users of spyware infested sites that may damage their users computers and a lot of other stuff in the SERP's, so engines are advising users about sites now days.

Should Google just list sites blocking nations with the disclaimer that "you may not be able to view this site, use a proxy" ????
Reply With Quote
  #46 (permalink)  
Old 07-03-2008, 02:15 PM
WebProWorld Veteran
 
Join Date: Jul 2004
Posts: 981
activeco RepRank 2
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by kgun View Post
... shall the posts on an extranet be index- and archivable?
OK I see the connection with the thread.

Extranets are usually closed behind firewalls so there is no access from general public as well as from outside bots.
However there is a thing called Google Search Appliance that provides Google technology for closed environments.
With it, an organization gets completely internal Google search engine.
__________________
Impossible? You just underestimate the time.
Reply With Quote
  #47 (permalink)  
Old 07-03-2008, 06:23 PM
kgun's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: May 2005
Location: Norway
Posts: 6,635
kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by crankydave View Post
Except that "users" are not assigned individual IP's. Blocking a particular IP or IP range prevents everyone in the "region" from access not just the individual.

Dave
And blocking an unique IP may block out a lot of sites if they are hosted on a shared server.
Reply With Quote
  #48 (permalink)  
Old 07-03-2008, 06:27 PM
kgun's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: May 2005
Location: Norway
Posts: 6,635
kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by Webnauts View Post
Already my team Information Architect is working on a script that blocks suspicious bots by IP for 24 hours, and then get unblocked. This thread is a good hint to extend that also for bad visitors. I will discuss that with him tomorrow to see what and how that would be possible.
I tried that manually without any success.

If you use a script, may be the best is to write a spider trap: Scripts: Spider Blocking :: .htaccess, PHP, Block Bad Bots with .htaccess or PHP

Last edited by kgun; 07-03-2008 at 06:37 PM.
Reply With Quote
  #49 (permalink)  
Old 07-03-2008, 06:56 PM
nullvariable's Avatar
WebProWorld Pro
 
Join Date: Sep 2005
Location: Greenville, SC
Posts: 216
nullvariable RepRank 1
Default Re: Google Says Blocking Countries Outside of the US is Against

Quote:
Originally Posted by kgun View Post
I tried that manually without any success.

If you use a script, may be the best is to write a spider trap: Scripts: Spider Blocking :: .htaccess, PHP, Block Bad Bots with .htaccess or PHP
Actually the script that I am writing will be directing bad requests and IPs to a Spam poison/spider trap part of the server. I discovered that someone wrote a really crappy crawler and was crawling my site with it. They were following links with ":" and it didn't even correctly remove the " at the end of some requests. Its pretty easy to setup a robots.txt and then drop in a link that can be spidered by bad bots. My script just takes hits to this link and blocks them and forwards them off to the spam traps/honey pots. It lowers the load on the main site and fills spammers databases with junk. But Googlebot and others that respect robots.txt will never run into all of this. Of course there are the actions of individual users to consider and one way I am thinking of doing it is to have all the error pages log hits and if someone hits an error page more than X times they get blocked for 24 hours.

Most of the IPs that I see doing this are in China, India and other locations, where I really could care less about blocking some guy and his neighbors for a few hours. The likelyhood that they are actually surfing my site is pretty slim. Basically I will be looking to filter unusual (or typical bad bot) behaviours and then simple block the IP for 24 hours and forward it off to a poison trap or honey pot.
__________________
Blog | Web Consulting
Reply With Quote
  #50 (permalink)  
Old 07-03-2008, 07:12 PM
kgun's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: May 2005
Location: Norway
Posts: 6,635
kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4kgun RepRank 4
Default Re: Google Says Blocking Countries Outside of the US is Against

Now I invite all of you that have posted in this thread to rethink the issue. Most of us start out by:

allow from all IP regions.

Why not turn that around and

deny from all

and gradually open up IP regions.
Reply With Quote
Reply

  WebProWorld > Search Engines > Google Discussion Forum

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Blocking IPs for Countries will it affect SEO Steven1976a Search Engine Optimization Forum 1 06-24-2008 08:21 PM
prices on google for countries brandyace Google Discussion Forum 2 06-12-2007 09:53 AM
Google pays more for clicks from 'big' countries...? Ricardo Zea Google Discussion Forum 19 10-10-2006 05:28 PM
Google results in foreign countries toby_bird Google Discussion Forum 9 02-03-2005 11:20 AM
Google Domains For 10 More Countries Now Available WPW_Feedbot Search Engine Optimization Forum 0 01-16-2005 03:30 PM


All times are GMT -4. The time now is 10:22 PM.



Search Engine Optimization by vBSEO 3.3.0