noindex, follow parameter

OpenYourEyes · #1 (**permalink**) 07-20-2004, 06:14 PM

Hello.

I have a sitemap that has been causing me some problems. First it kept getting ranked above the content pages that in linked to. So instead of having good titles & descriptions for my search engine listings, it would be 'Sitemap'. I added a <META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW"> tag to that page. It worked great in Google, but Yahoo seems to be having problems with that tag. The page does not get indexed by Yahoo, but the spider is not following the links. So now in Yahoo, not only does our sitemap not get listed, but all of the linked pages do not get indexed either.

Has anyone else had a similar problem? Or does anyone know a way around this? Would there be a way to apply the robots tag to only the Google spider?

ronniethedodger · #2 (**permalink**) 07-21-2004, 05:10 AM

I have not had that experience before. But personally, I would remove the No Index part of that meta and put it back to Index. Having one page up in the results is better than having none and for some reason Yahoo seems to think that your sitemap is the page to be listed. Since you have not left a link to your website, I cannot give you any thoughts on to why it is doing that ... leave a link next time.

OpenYourEyes · #3 (**permalink**) 07-21-2004, 01:18 PM

In Google the noindex, follow parameter works perfectly. We're in the top 3 for almost everyone of our search phrases. By removing the noindex and having the sitemap show up for all of these searches, we would be losing a lot of click throughs. The pages a level down from the sitemap have targeted titles & descriptions that relate to exactly what the person searched for. By having a generic sitemap title & description, people will instead click on the other links that relate to exactly what they want. The sitemap also doesn't have the keywords in the title, so it would rank lower than the targeted pages. The sitemap just gets picked up first because it is fewer clicks away from the frontpage and it also has more links to it. When we added the noindex parameter, our Google rankings improved to where they normally are near the top. Now Yahoo doesn't rank us well.

Can anyone else help?

ronniethedodger · #4 (**permalink**) 07-21-2004, 05:43 PM

Is there a reason why you cannot leave a link to the site? Like I said, I have not experienced this on any site or know of any others that have. Without visually seeing the pages in question, nobody will be able to give you an answer without first looking at it.

OpenYourEyes · #5 (**permalink**) 07-23-2004, 08:35 PM

I choose not to give out the URL. On this forum people are often more critical than they are helpful. The problem is just in that one line of code, so sharing the url wouldn't help much. It's alright though. Someone did help me on a different forum. The link popularity of the sitemap was too high, so once I remove the sitemap link on every page it should be fine.