noindex, nofollow - index, follow

[EJRS.COM]

Keep Your Web Site Out Of The Search Engines
<meta name="robots" content="noindex,nofollow">

Keep any pages out of search engines by sticking this simple code within the metatags.


If you want any pages to be seen by the bots,
<META NAME="robots" CONTENT="index, follow">
<META NAME="GOOGLEBOT" CONTENT="INDEX, FOLLOW">

If you need more infogo to http://ejrs.com/webmasters.html

If I am online now and you can click on the Live Help Button and I can help you out with more queries. If I am offline, leave a message by clicking on the live help button (It will say Unavailable, if I am not online). I will get back to you soon as possible. Have a great day.

[Mel]

Hi EJRS

Yep but why would you want to keep your pages out of the search engines?

You can also do the same thing on a larger scale with the robots.txt file.

[cvos]

I suggest you put javascript into your frames to force the frameset.

the better question is why would one ever want to use frames? it breaks the user experience.

http://www.useit.com/alertbox/9612.html

[Duncan Pollock]

ejrs: I think I've got this right:
My website includes two pages, one thanking someone when they sign up to join my mailing list and one asking them to Go Back if they haven't filled in the form with their e-mail address. So these two pages could usefully have the <meta name="robots" content="noindex,nofollow"> code added to them, yes?
And, to be safe rather than sorry, adding this code to the two pages but leaving all my other pages as they are is AOK? Or is it necessary -- or advisable/wiser -- to add the <META NAME="robots" CONTENT="index, follow"> and <META NAME="GOOGLEBOT" CONTENT="INDEX, FOLLOW"> code to them? (Furthermore, are both the "robots" and "Googlebot" lines required, or does just the "robots" one suffice?)
Sorry to ask so many For Free questions!

Thank you
Duncan

PS I'm NOT complaining but I can't help wondering why we're dealing with this topic under the heading of Inktomi? Should it perhaps be moved by one of our moderators to one of its own?

[minstrel]

Originally Posted by Duncan Pollock

My website includes two pages, one thanking someone when they sign up to join my mailing list and one asking them to Go Back if they haven't filled in the form with their e-mail address. So these two pages could usefully have the <meta name="robots" content="noindex,nofollow"> code added to them, yes?

Yes... or you could exclude those pages using a robots.txt file in the root directory of your website, thus:

User-agent: *
Disallow: thankyoupage.htm
Disallow: gobackpage.htm

These lines tell all spiders to ignore those two pages and index everything not specifically "disallowed".

To provide specific instructions for googlebot that are different from your general instructions to spiders, use this:

User-agent: Googlebot
Disallow: thankyoupage.htm
Disallow: gobackpage.htm

And, to be safe rather than sorry, adding this code to the two pages but leaving all my other pages as they are is AOK? Or is it necessary -- or advisable/wiser -- to add the <META NAME="robots" CONTENT="index, follow"> and <META NAME="GOOGLEBOT" CONTENT="INDEX, FOLLOW"> code to them?

The default behavior is "index, follow" - however, it doesn't hurt to include that tag on each page - it's only a few characters...

(Furthermore, are both the "robots" and "Googlebot" lines required, or does just the "robots" one suffice?)

The "robots" one alone is sufficient. You only need to include instructions to Googlebot if those instructions are different from the instructions you want to give to other robots. From Google's own website:

If you want to prevent all robots from indexing individual pages on your site, then you can place the following meta tag element into the page's HTML code:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

If you want to allow other robots to index individual pages on your site, preventing only Google's robots from indexing the pages, use the following tag:

<META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">

PS I'm NOT complaining but I can't help wondering why we're dealing with this topic under the heading of Inktomi? Should it perhaps be moved by one of our moderators to one of its own?

Excellent point! Thanks, Duncan - sometimes it's hard to remember what the beginning of a thread was once you get engrossed in the ongoing discussion - consider it done! well, because actually, now it IS done :o)

[EJRS.COM]

Minstrel You Crack Me Up. I was trying to figure out when on earth I made a NEW TOPIC called No Index No Follow. Now I see I was moved :) Thanks Minstrel. It's cool to have a cool dude like yourself for a mod.

Duncan you answered yourself very well on that question and Minstrel helped to clarify things a whole lot.

Hey Mel, good to "meet" someone else from Malaysia. Duncan and Minstrel kinda answered the question for me :) Ain't this forum cool?

[Duncan Pollock]

ejrs and minstrel:
Thanks a bunch. And, ha, ha, please note that I've put the correct wording against the Post subject:
But seriously, I do agree, ejrs, about how useful the whole WebProForums are. I've learned a great deal from them, as, for instance, in this (no)index,(no)follow idea.
You're quite right, too, about the cool dudes like my fellow Canuck minstrel.

Cheers to the twose of youse!

Duncan

[minstrel]

Thanks, ejrsdotcom and Duncan...

It's been one of those days - I needed to find your posts today...

:o)

[sudhani]

Hi, Minstrel,

I always wonder whether the following tag does exist?
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">

Is there any place where i can find authentic tags apart from w3c?

Sudha

[minstrel]

Hi, Sudhani:

Here's some starting points - bear in mind that some of the meta tags are either pretty much archaic or are special purpose only and that the majority aren't very useful for search engine optimization.

Back to Basics: Meta Tags (old article)


A Dictionary of HTML META Tags

META FAQ

[MoZarko]

Hi all,
I've been searching the formun but couldn't find an answer to my question (it's probably there). I would like your opinions on which you think to be the best in the meta tag: index, follow or all or index, follow, all? Plus, do you need the revisit after tag or is this no bigging? I appreciate your thoughts and opinions. Thanks...

[paulhiles]

Originally Posted by MoZarko

Hi all,
I've been searching the formun but couldn't find an answer to my question (it's probably there). I would like your opinions on which you think to be the best in the meta tag: index, follow or all or index, follow, all? Plus, do you need the revisit after tag or is this no bigging? I appreciate your thoughts and opinions. Thanks...

Hi there MoZarko,

there's not really such a thing as 'the best' way to use a meta tag. You optimise the tags according to how you wish your site to be indexed by search engine bots. There's a recent discussion here on this very page:
http://www.webproworld.com/viewtopic.php?t=12535

[minstrel]

You probably don't NEED either but it won't hurt to include "index, follow".

It's probably best to omit "revisit after" - if the spiders pay any attention to it at all, it may mean they will visit you less frequently than their own schedules...

I'm going to move your post to the previous thread discussing this issue.

[paulhiles]

Originally Posted by minstrel

It's probably best to omit "revisit after" - if the spiders pay any attention to it at all, it may mean they will visit you less frequently than their own schedules...

Yeah agreed Minstrel, this point about the "revisit-after" instruction was discussed in another thread previously.
http://www.webproworld.com/viewtopic.php?t=1579

HTH!

Paul

[MoZarko]

Thanks. Appreciate your opinions.