An Ethics Question...

[wige]

Hi everyone.

I recently came across a technique that I believe could be used to trigger duplicate content penalties against almost any web site. This could, at least in theory, be used to affect a web site's rankings, either accidentally or maliciously. I would like to open a discussion on the topic to get an idea of whether such a practice would actually be effective, and if so, to try to come up with ways this technique could be detected, prevented, and recovered from.

My dilemma is that in order to discuss this technique, which would definitely be black hat, I would have to give information that would allow people to utilize the technique fairly easily. I try to be careful when describing attacks and vulnerabilities to avoid giving so much information that my posts become instruction manuals, but in this case I am not sure I would be able to discuss the issue without being so detailed.

So, what should I do? Should I post the information about this technique, which might be abused but which we might come up with a way to prevent or defend against? Or should I err on the side of caution and not post the information?

[james113]

In my opinion, discussing this issue on a public thread is not ideal. It would indeed enable people to exploit it. I would suggest inviting known white hatters and trusted individuals to participate in a discussion group via a private chat room or forum.

[Dubbya]

I concur.

Were you to post the information, every script kiddie in here would be l33tly h@x0ring site rankings in seconds.

Keep it on the down-low.

[seopo]

I'm not recommending either but security through obscurity only lasts so long.... besides be very hard to use dupe content penalty to a correctly optimised site?

[netman4ttm]

It really depends on how obscure you thing this issue is.

I would advise the search company/s of the issue and wait for a response.

After a reasonable amount of time (your decision) I would publish.

Since this is not a security issue but a search problem I don't see hackers going wild with it. I mean how much fun is Beavis going to have trying to mess up IBM's page rank?

[HHI_Golf_Guy]

According to Google they do not impose a duplicate content penalty (see the second paragraph):

Official Google Webmaster Central Blog: Demystifying the "duplicate content penalty"

Yahoo and MSN might have a dupe penalty, but for many those search engines are now irrelevant anyway.

[Tubby]

Wige said
"I would like to open a discussion on the topic to get an idea of whether such a practice would actually be effective"

Do it Wige, We have a nice mixture of grown ups and kids here. The best defence is is knowledge.

Or simply wait for three months when two million webmasters are already testing and practising the technique.


Web Rankings change daily. Duplicate content penalties? Nobody at WPW has any power to issue penalties.
Are we really that worried and concerned that Google might get "tricked".

CONTENT IS KING . create some wige
I thought moderators were encouraged to start threads that would attract traffic.

[Big Juice]

Post and discuss away. The only sites that should fear something like this would be the content scrapers anyway so lets have at her unless of course you like seeing scrape sites ranking in the search engines.

[dwells87199]

Open it up, let others exploit it so that it can be identified and fixed.

[danlefree]

One would hope that this technique is not something simple like... linking to the non-canonical link, adding irrelevant get and anchor tags to URL's pointed at the domain (like Urchin/GA variables), providing the domain's IP address (every webserver has an IP, after all), et cetera.

Each of these methods will create a genuinely "different" URL, though I honestly doubt that penalties are attributed in most cases.

The "penalty" for having multiple URL's point at the same content is the dilution of the content's ranking under a single URL. Search engines like MSN and Yahoo will still list each link (if memory serves), though the content will have multiple links of lesser weight than a single link would... but if you're spamming competitors' links in an attempt to get them penalized, you are essentially contributing to the greater likelihood that the links which you are providing are helping their content's ranking under different URL's...

[Feydakin]

In theory this, whatever it is, would only work against a new website vs one that is established.. As was pointed out, Google claims to have the duplicate issue under control so an established site should see no harm, while the dupe should see no benefit..

At least that's what I've been testing the last couple of months.. So far it 'looks' like Google has a handle on it..

[MrGamm]

Originally Posted by wige

So, what should I do? Should I post the information about this technique, which might be abused but which we might come up with a way to prevent or defend against? Or should I err on the side of caution and not post the information?

If people don't find what they are looking for the first time. What makes you think they are going to find what they are looking for after the 2nd,3rd,4th or fifth time they see the exact same content they just bounced from?

If it works and it returns more visitors and converts more sales or generates business. Make money with it.

If you have the ability to take someone else's website out... Contact the search engine people directly with your concerns. If it is Google... You might have a hard time contacting anyone on staff... Why not go over to the Google Webmaster Groups and post your concerns there? They have staff who review the posts. Providing details there will ensure that at least some of the right people get a chance to review the material.

http://www.google.com/support/forum/p/Webmasters?hl=en

Publishing instructions on how to exploit software vulnerabilities is not a good idea. Many people will ban you from their forums if you do. Nobody is going to stop you if you post the instructions on your own site or a site which encourages that type of material like some security boards, or hacker sites. In the case of Google... I don't think you would have much choice except to post in one of their public areas if you wanted to do the ethically right thing... jmo...

Typically... you can't harm another persons website in the rankings unless you have access to their websites content.

[ddwalker1]

Seems to me that an open discussion would allow participants to guard against such problems. Since we're only talking about search engine ranking, I'm not sure the information would be as interesting to 'hackers' as you might think. If you didn't want to discuss it, why did you bring it up? : )

[ron angel]

Originally Posted by wige

Hi everyone.

I recently came across a technique that I believe could be used to trigger duplicate content penalties against almost any web site. This could, at least in theory, be used to affect a web site's rankings, either accidentally or maliciously. I would like to open a discussion on the topic to get an idea of whether such a practice would actually be effective, and if so, to try to come up with ways this technique could be detected, prevented, and recovered from.

My dilemma is that in order to discuss this technique, which would definitely be black hat, I would have to give information that would allow people to utilize the technique fairly easily. I try to be careful when describing attacks and vulnerabilities to avoid giving so much information that my posts become instruction manuals, but in this case I am not sure I would be able to discuss the issue without being so detailed.

So, what should I do? Should I post the information about this technique, which might be abused but which we might come up with a way to prevent or defend against? Or should I err on the side of caution and not post the information?

You remind me of a friend who has some technical knowledge that said to me in all seriousness that he had invented a mobile phone battery that never has to be recharged but is afraid to tell anybody in case they steal the idea will not even tell a patent agent. I am an electronics design Eng by trade and I could tell him in a couple of minutes if it was viable.about ten years later he has done nothing. ( he was right of course I would have gone into partnership with him developed the idea made loads of money but given him the credit for the idea) I have another elderly friend who is very rich he is a watchmaker that can fix any watch that others cant. I have seen him make parts for antique watches from a piece of metal & handled a clock he repaired for a museum that belong to Napoleon & other objects of art. He has invented a watch like device for charging battery's by movement for use in pacemakers, which I gave him advise on the electrical side for a prototype he built that works but needs a lot of development.
he told a fellow business friend who sold the idea to a large drug company in the USA. problem for his friend that a professor from a London university had seen it & placed an article about it on a website my friend is now in the process of suing his friend with the help of a member of parliament from UK government, not only for the money, but so that people know that he invented the device which to him is more important & as he has the only one in existence for damages if it is ever produced without his agreement. also the man who sold the idea for fraud.
which brings me to the point of all this,If you don't tell anybody about it then there is no proof the idea works as it cant be tried in the field but if you do somebody will steal & use the idea.best is to tell everybody how it works so they know it was you who discovered it, or sell the idea to to an seo company making lots of money until it is found & blocked by google. over to you.

[getmea]

Seems like half of us are against the post while the other half are interested...myself, I am interested in the technique. In terms of intranet coding, content replication would be allowed and encouraged in big IT shops like mine.

[james113]

Originally Posted by Tubby

Wige said
"I would like to open a discussion on the topic to get an idea of whether such a practice would actually be effective"

Do it Wige, We have a nice mixture of grown ups and kids here. The best defence is is knowledge.

Or simply wait for three months when two million webmasters are already testing and practising the technique.


Web Rankings change daily. Duplicate content penalties? Nobody at WPW has any power to issue penalties.
Are we really that worried and concerned that Google might get "tricked".

CONTENT IS KING . create some wige
I thought moderators were encouraged to start threads that would attract traffic.

It appears to me that this thread is attracting traffic. I have been reading posts in this forum for 9 years, and I would lose a great deal of respect for this forum if moderators published blackhat methods that work. There are just too many kids around. Lets be realistic here!

According to Google they do not impose a duplicate content penalty (see the second paragraph):

Official Google Webmaster Central Blog: Demystifying the "duplicate content penalty"

Yahoo and MSN might have a dupe penalty, but for many those search engines are now irrelevant anyway. .

You should read all the paragraphs and follow the suggested links as well. This is the second post in WPW That I have seen today refering to this Google blog post, by someone who doesn't seem to understand it. There are indeed duplicate content penalties that are in effect.

Lets just say that only 20% of the people online use Yahoo and MSN for searching. 20% of a few hundred million is still a very large number of possible users/customers that I myself do not consider irrelevant!

[kgun]

Originally Posted by wige

I recently came across a technique that I believe could be used to trigger duplicate content penalties against almost any web site.

I doubt that.

So we should not discuss openly how to protect our home. I agree with Tubby.

It is not dangerous when it is open. It is dangerous when the technique is hidden. May be some SE companies will thank you too.

[kgun]

Proposal:

Email a link to this thread to the most important native SE's before posting the method, unless we will suspect you for segregation

[Dubbya]

In retrospect, perhaps our more liberal minded friends are right. I'm sure they'll be just as eager to have us test this nefarious technique on their sites first.

Don't play favorites! Please post detailed instructions so that everyone can join in the fun.

.02

[Webnauts]

Originally Posted by james113

There are just too many kids around. Lets be realistic here!

I fully agree. And a lot of wannabe SEO experts or gurus (as they call themselves). An open discussion on such a topic here, can only trigger more junk and trouble in our industry.

[ron angel]

Originally Posted by Webnauts

I fully agree. And a lot of wannabe SEO experts or gurus (as they call themselves). An open discussion on such a topic here, can only trigger more junk and trouble in our industry.

we have no proof that this "method" even exists let alone works.
how about the "experts" emailing each other coming to a conclusion and letting the rest of us know their learned judgement.... soon
I wait all expectorant with baited breath.

[Webnauts]

Originally Posted by ron angel

how about the "experts" emailing each other coming to a conclusion and letting the rest of us know their learned judgement.... soon

That is a totally different story. We release stuff when we are sure that there will not be any harm.

[Dubbya]

Originally Posted by Webnauts

I fully agree. And a lot of wannabe SEO experts or gurus (as they call themselves). An open discussion on such a topic here, can only trigger more junk and trouble in our industry.

You did detect the note of sarcasm in my reply, right?

[Webnauts]

Originally Posted by Dubbya

You did detect the note of sarcasm in my reply, right?

I sure did!

[janeth]

Originally Posted by james113

It appears to me that this thread is attracting traffic. I have been reading posts in this forum for 9 years, and I would lose a great deal of respect for this forum if moderators published blackhat methods that work. There are just too many kids around. Lets be realistic here!

It's okay to publish methods that don't work but if they work you’ll lose respect for the forum.

What type of respect do you have for the forum?

Originally Posted by james113

There are indeed duplicate content penalties that are in effect.

I'm sure there are but they don't work like you and others seem to think they do.

[kgun]

I have always learned that underground methods are more dangerous when they are hidden.

In essence, Wige says, that he has a method that the SE algorithms can't fight (or don't already know).

I still, highly doubt that.

Send an ePost to the SE's explaining the method.

Wait some days (weeks) and publish the method.

Google:

password cracker (you find it in Google suggest).

Use Google:

translate

look up a forum in farawayland and you find methods that can occupy you for the rest of your life.

opera:config

So I should not inform about the concequences to a newbee of hidden text (that is know to the SE's) and other methods.

Until the opposite is proved, this is rediculous IMO.

Should I not tell this?

[crankydave]

Originally Posted by kgun

In essence, Wige says, that he has a method that the SE algorithms can't fight (or don't already know).

hmmmmmm... Not sure that's what's being said at all. Just that he believed it to be possible and whether or not it was wise to bring it up publicly.

Don't wish to step on wige's toes here

FTR... It is and has been being discussed. In a nutshell, it involves a proxy hacking/hijacking/scraping. As I'm sure many of you are aware this was a problem some time ago that Google had quite a bit of difficulty with but has already resolved it and to the best of my knowledge remains resolved.

The twist though, was it was targeting "new content" rather than existing content attempting to get the duplicate indexed before the original.

IMO, theoretically possible BUT there would be a whole lot of qualifiers, a very narrow range where it MIGHT have an effect, and a whole lot of work for something that could likely reap nothing in return and could easily trash the exploiter.

Dave

[kgun]

Originally Posted by crankydave

hmmmmmm... Not sure that's what's being said at all. Just that he believed it to be possible and whether or not it was wise to bring it up publicly.

Don't wish to step on wige's toes here

I am one of the people giving much credit to Wige. I often have to distribute more green color before I can give him additional.

Ok, if he assumes that it is better that the method is only (I also highly doubt that) known to the SE's, then send them an eMail.

Do some of you remember when DVD John cracked Google's video code in a day or two. It impressed the world.

One of the strengths about the USA is in my opinion that the discussion is so open there.

[Webnauts]

Wige, if you implement the following, do you think the method you are talking about will be safe? Can you give a little hint?

RewriteEngine On
Options +FollowSymLinks
ServerSignature Off
RewriteCond %{REQUEST_URI} ^/(,|;|:|<|>|”>|”<|/|\\\.\.\\).{0,9999}.* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(;|<|>|’|”|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|cast|set|declare|drop|update|md5|benchmark).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(<|>|’|%0A|%0D|%27|%3C|%3E|%00).* [NC]
RewriteRule ^(.*)$ http://www.letsgoforitwige.com

### Safe Request Methods - Denies any request not using GET,PROPFIND,POST,OPTIONS,PUT,HEAD ###
#RewriteCond %{REQUEST_METHOD} !^(GET|HEAD|POST|PROPFIND|OPTIONS|PUT)$ [NC]
#RewriteRule .* - [F,NS,L]

### HTTP PROTOCOL - Denies any badly formed HTTP PROTOCOL in the request, 0.9, 1.0, and 1.1 only ###
RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ .+\ HTTP/(0\.9|1\.0|1\.1) [NC]
RewriteRule .* - [F,NS,L]

RewriteCond %{HTTP_USER_AGENT} libwww [NC,OR]
RewriteCond %{QUERY_STRING} ^(.*)=http: [NC]
RewriteRule ^(.*)$ - [F,L]

# Block proxy servers from site access
RewriteEngine on
RewriteCond %{HTTP:VIA}                 !^$ [OR]
RewriteCond %{HTTP:FORWARDED}           !^$ [OR]
RewriteCond %{HTTP:USERAGENT_VIA}       !^$ [OR]
RewriteCond %{HTTP:X_FORWARDED_FOR}     !^$ [OR]
RewriteCond %{HTTP:PROXY_CONNECTION}    !^$ [OR]
RewriteCond %{HTTP:XPROXY_CONNECTION}   !^$ [OR]
RewriteCond %{HTTP:HTTP_PC_REMOTE_ADDR} !^$ [OR]
RewriteCond %{HTTP:HTTP_CLIENT_IP}      !^$
RewriteRule ^(.*)$ - [F]

or/and

<base href="http://www.letsgoforitwige.com">

[janeth]

Originally Posted by crankydave

The twist though, was it was targeting "new content" rather than existing content attempting to get the duplicate indexed before the original.

You would have to be on your toes.

[janeth]

Originally Posted by Webnauts

Wige, if you implement the following, do you think the method you are talking about will be safe? Can you give a little hint?

If we are targeting new content that wouldn’t make any difference?

The question mark because I'm guessing.

[james113]

Originally Posted by janeth

It's okay to publish methods that don't work but if they work you’ll lose respect for the forum.

What type of respect do you have for the forum? .

I never said it was ok to publish any blackhat methods in this forum whether they work or not. There are plenty of hacker sites for that kind of crud! I'm sorry, but I and other legitimate webmasters come here to learn and share legitimate info.

Type of respect? Didn't know that different types existed. Are you asking how much I respect this forum? Enough to keep coming back for over 8 years.

Originally Posted by janeth

I'm sure there are but they don't work like you and others seem to think they do.

How do you know what I think or know about dup. content issues, as I have never elaborated on the subject in this forum or anywhere else for that matter.

[janeth]

Originally Posted by james113

I never said it was ok to publish any blackhat methods in this forum whether they work or not. There are plenty of hacker sites for that kind of crud! I'm sorry, but I and other legitimate webmasters come here to learn and share legitimate info.

I was just playing with you however, you did say that it was not okay to post black hat stuff that works. Which would mean that it was okay to post stuff that didn't work.

Originally Posted by james113

Type of respect? Didn't know that different types existed.

There are all types of respect.

Originally Posted by james113

but I and other legitimate webmasters come here to learn and share legitimate info.

And I am one of those.

Originally Posted by james113

How do you know what I think or know about dup. content issues, as I have never elaborated on the subject in this forum or anywhere else for that matter.

I had a pretty good idea from your statement.

Originally Posted by james113

There are indeed duplicate content penalties that are in effect.

That would not be true.

[james113]

Could you please give me just 2 examples of respect "types"?

There are indeed duplicate content penalties that are in effect.

How did this statement give you a pretty good idea that I don't know how dup. content issues seem to work?

[wige]

Thanks for the feedback everyone. Honestly, I think this thread ended up a lot more interesting than a discussion of the technique would have been...

Originally Posted by Webnauts

Wige, if you implement the following, do you think the method you are talking about will be safe? Can you give a little hint?

I think that would prevent it fairly effectively.

There are two parts to the issue. One is the use of CGI Proxies to make the duplicates (which your .htaccess code should prevent, mitigating the issue) and the other is using a method to trick search engines into crawling the created duplicates first.

My thinking with this was that smaller sites might be vulnerable to this type of attack if multiple links to the duplicate content could be seeded on web pages just before they got scanned by Googlebot, in an attempt to make Google think the original is a later copy of the duplicate. It is something of an edge case, unlikely to be effective, but the possibilities did make me curious.

Right now, I think the CGI proxies themselves are the biggest issue, both because they are the central issue in attempting this method, and for the reason that even if it doesn't impact SEO, webmasters likely would want ways to block their data from being reproduced on these sites. I am going to continue researching the topic, and will possibly put together a writeup in the near future about detecting and preventing this type of duplication, including solutions like what Webnauts posted above.

As far as the other part of the technique, predicting spider crawls and exploiting that information, I am not going to go into detail on that in a public way. Even if it may not be an issue here, I can think of a few possible ways it could be abused, so I will keep that private.

[james113]

There may be others, but the only dup. content penalty that I know for a fact to exist is multiple indexed URLs serving up one website, so there are indeed duplicate content penalties that are in effect

[janeth]

[QUOTE=james113;412269]Could you please give me just 2 examples of respect "types"?

Respect for the safety of others.

Respect for property

Respect for your parents

Originally Posted by james113

How did this statement give you a pretty good idea that I don't know how dup. content issues seem to work?

Because there is no penality.

[Tubby]

James 113 asks '
"Could you please give me just 2 examples of respect "types"?"

My Father told me in regard to the Police "Always respect the uniform son" Of course we all understand that the person wearing the uniform could well be a wanker. A cautionary respect - Fear.

I have placed the words on threads in WebProWorld. (To Webnauts for instance) - "I have always respected You. "This is a different type of respect. . I do not always agree with what he might write. But I will bow to his greater knowledge in certain fields. I would not argue with any statement he makes unless it strays beyond his field of knowledge. Inside that narrow field . . I hold a High regard respect - Submissive

I understand these to be different forms of respect. They have a different emotional source, So I personally feel that they should qualify as a different form of respect.

[janeth]

Originally Posted by james113

There may be others, but the only dup. content penalty that I know for a fact to exist is multiple indexed URLs serving up one website, so there are indeed duplicate content penalties that are in effect

I've got 2,230 indexed URLs on Geeks on Steroids that all serve up my website. If that is a penalty then it's a darn good one.

[crankydave]

Originally Posted by james113

There may be others, but the only dup. content penalty that I know for a fact to exist is multiple indexed URLs serving up one website, so there are indeed duplicate content penalties that are in effect

I'm curious. How do you know this is a "penalty" and not a "filter"? Which gets "penalized"? All of them? Under what circumstances? How is it decided? What exactly is the "penalty"?

Dave

[Webnauts]

There is no duplicated penalty! That is a filter. But there are a lot of examples out there, that the filters do not always work properly.

[Webnauts]

Originally Posted by janeth

I've got 2,230 indexed URLs on Geeks on Steroids that all serve up my website. If that is a penalty then it's a darn good one.

Google returns about 2960 pages site:geeksonsteroids.com - Google Search and in the main index are only about 302 pages.

So you have 2658 pages in the supplemental index. Duplicated content? You should know that. If that is the case, such pages are moved to the supplemental index.

[crankydave]

Originally Posted by Webnauts

Google returns about 2960 pages site:geeksonsteroids.com - Google Search and in the main index are only about 302 pages.

So you have 2658 pages in the supplemental index. Duplicated content? You should know that. If that is the case, such pages are moved to the supplemental index.

And for Seo Workers Google returns about 149 pages and in the main index are about 30. Duplicated content? Something else?

Dave

[Webnauts]

Originally Posted by wige

Even if it may not be an issue here, I can think of a few possible ways it could be abused, so I will keep that private.

Do you mind sharing with me?

[james113]

Thanks for clearing up the respect issue tubby.

All I know is that a friend of mine has a website(yahoo store) that was indexed in google. He registered a dozen new domain names that consisted of phrases that were relevant to his products. Before he figured out how to redirect the domains to the original url, somehow these urls(duplicate websites) got indexed by google. When these urls were indexed by google his site instantly went to the top of the search results for each and every new phrase represented by the new urls for about a week, then his main url suddenly lost its search results positions that it had maintained for several months prior. His main site url, once at position one in the search results for several phrases could no longer be found on page one for any search phrases. All the newly indexed urls lost there first page positions as well and google began to very slowly unindex the new urls after he reported it to google through webmaster central. It took almost 3 months for his original url to regain its original positions in the search results. This all happened in the last 6 months and it sure seemed like a penalty to me.

[Webnauts]

Originally Posted by crankydave

And for Seo Workers Google returns about 149 pages and in the main index are about 30. Duplicated content? Something else?

Dave

It is something else. SEO Workers have in total 34 pages. The rest number of pages are cached pages of our forums which do not exist any longer.

Evidence: The forums were on forums. seoworkers .com and now they are on www. seoworkers. net. And in addition on two different unique IPs. Try to make a site: search for seoworkers.com and tell me how many search pages deep can you go. Google was and is still claiming that they update their supplemental index more frequently, so we should not really worry about the supplemental pages that are no more labeled. But as you see they are telling BS.

Or do we have a third index called Cache?

[wige]

Originally Posted by crankydave

I'm curious. How do you know this is a "penalty" and not a "filter"? Which gets "penalized"? All of them? Under what circumstances? How is it decided? What exactly is the "penalty"?

Dave

Well, I suppose if my site was filtered, I'd be calling it a penalty...

[Webnauts]

Originally Posted by crankydave

And for Seo Workers Google returns about 149 pages and in the main index are about 30. Duplicated content? Something else?

Dave

By the way thanks for pointing out this issue. I began taking care of it. I think it is time for a PR6 again.

[james113]

Originally Posted by wige

Well, I suppose if my site was filtered, I'd be calling it a penalty...

Is this a wise crack towards me wige? Are you saying that what happened to my friends website was a filtering process and not a penalty?

[kgun]

Outstanding sites (mini networks) don't need a search engine (or SEO). They will be known by word of mouth.

Paradox: Yahoo paved the way for Google?. I first were told about Google.

You can not destroy the Coca Cola brand. You can not destroy an excellent site by throwing negative SEO against it. People will simply look it up.

Marketing is a type of drama. (Scroll down).