Question About Robots and Adwords

dsellers · #1 (**permalink**) 03-05-2008, 04:11 AM

I’ve been searching the forum and found any info about this question. From what can tell, Google penalizes organic rankings for duplicate content. Our main site is being designed for SEO (we use index/follow tags). But we have several AdWords minisites that we use that have duplicate content of the main site (landing pages are modified a bit according to AdWords keywords). For these types of AdWords minsites, I’m wondering what would be the best index/follow tag to use? I don’t want the duplicate content on the minisites to compete/pentalize our main website, but I also want the AdWords spiders to make full use of them (to keep my AdWords Landing page quality score up, etc). Is it suggested to use noindex/follow on these minisites so that the AdWords will index them, but the Search Engines won’t try to index them and see them as duplicate content? Or is it safe to use noindex/nofollow so that the engines will see that we are not trying to create duplicate sites, but will this tag still be safe to use with AdWords. Any suggestions would be greatly appreciated.

**incrediblehelp** · #2 (**permalink**) 03-05-2008, 09:26 AM

Quote:

Originally Posted by dsellers

I’ve been searching the forum and found any info about this question. From what can tell, Google penalizes organic rankings for duplicate content.

Nope they just filter it.

Have you tried blocking the bots from the minisites through your robots.txt file?

dsellers · #3 (**permalink**) 03-05-2008, 01:11 PM

Would this be the recommeneded meta tag to block robots on the adwords landing pages (I'm still new to this):

<meta name="robots" content="noindex,nofollow" />

bobitza · #4 (**permalink**) 03-05-2008, 03:25 PM

if you want to block robots on a page ... add it to robots.txt

your code will work with some robots and will not work with others.

activeco · #5 (**permalink**) 03-05-2008, 06:07 PM

Quote:

Originally Posted by dsellers

Would this be the recommeneded meta tag to block robots on the adwords landing pages (I'm still new to this):

<meta name="robots" content="noindex,nofollow" />

It seems like a simple question, but I'm afraid the answer is not so easy.
Sure approach would be robots.txt with:

User-agent: *
Disallow: /adwords-directory/

Adwords quality robot does NOT obey '*' clause, but if you wish, you can explicitly allow it:

User-agent: *
Disallow: /adwords-directory/

User-agent: AdsBot-Google
Disallow:

deepsand · #6 (**permalink**) 03-05-2008, 07:06 PM

Quote:

Originally Posted by dsellers

But we have several AdWords minisites that we use that have duplicate content of the main site (landing pages are modified a bit according to AdWords keywords).

Define "mini-sites."

Do you mean separate domains, sub-domains, or merely sub-directories?

dsellers · #7 (**permalink**) 03-05-2008, 10:38 PM

Activeco: I did a google search on “Disallow: /adwords-directory/” and this post is the only thing that came up – wow, that was fast. Do you know where I can find out more about customizing my robots tags for the purposes I have outlined here? I’m not here to needle anyone to death, I just need to figure out how to set my sites up correctly. I’m surprised there isn’t more info that’s easy to find regarding this –it’s seems like many people probably have the same issue. I would even pay for consulting time if I knew I was getting valid information.

Deepsand: The minisites are on separate domains. They are almost exactly the same, except for minor changes in the headlines, etc. Many of them are even identical. I have the same sales page on my main site as I do in the minisites. I am working to get the sales page on the main site better optimized for SEO, but I don’t care about SEO for the minisites (they are only for AdWords ads). This is why I’d like to cut the minisites off from robots so I don’t get penalized on my main site for duplicate content (my main site is the also the one that I want to show up in organic searches).

Thanks for any insights.

dsellers · #8 (**permalink**) 03-05-2008, 11:24 PM

I was just behind the times when it comes to robots.txt. I didn't know about it until today (I thought the meta tags controlled these things). Anyway, I found this help page:

How do I block or allow Googlebot?

Based on this, I think this what I should put in the robots.txt file of my ministes:

User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Disallow: /
User-agent: Googlebot-Image
Disallow: /
User-agent: *
Disallow: /robots.txt

This will allow the google adwords bot in (so I won't be penalized for adwwords quality score), but it will keep other bots out so I won't be penalized for duplicate content in relation to SEO. I put Disallow: /robots.txt line so that the robots.txt won't get indexed and cached. If I'm not understanding any of this correctly, please let me know.

deepsand · #9 (**permalink**) 03-05-2008, 11:44 PM

Quote:

Originally Posted by dsellers

I was just behind the times when it comes to robots.txt. I didn't know about it until today (I thought the meta tags controlled these things).

Actually, you are correct; the meta tags are intended to be used as you believe.

However, there is no way to force any spider to comply with either the meta tags or the robots.txt directives; some observe both, some one or the other, and some none at all!

Therefore, if it's of import to you, it's best to use both, and hope that the spider(s) in question will observe at least 1 of the 2.