Robot Text Files - DO Not Follow Sprecific Links

[markvredeveld]

I am trying to write a robot text file that allows all pages to be followed and indexed by robots but I do not want robots to follow or index any link on any page that starts with the travel now . com -- global discount hotel reservations url or with the travel now . com -- global discount hotel reservations URL. Would the robot text file below accomplish that? Simply stated, I do not want robots to follow links that start with the URLs listed below:


User-agent: *
Disallow: travel now . com -- global discount hotel reservations
Disallow: travel now . com -- global discount hotel reservations

[wige]

No.

A robots.txt file only applies to the domain (or sub-domain) that it is installed on. In other words, you would need to have a robots.txt file at reservations.5bestvacationrentals.com and reservation.5bestvacationrentals.com that denies everything (Disallow: * I think) and another robots.txt for your other domains that allow everything.

[markvredeveld]

Is there no way to diect a robot from following a link?

[markvredeveld]

Is there no way to stop robots from following a specific link on your website page?

[wige]

Yes, there are various ways of telling robots how to handle links. However, if you have multiple domains or subdomains, each one needs it's own robots.txt file.

[blitzen]

A Standard for Robot Exclusion

The robots.txt file is used to restrict bots from certain pages/directories.*

To get them to follow links, there's now a robots.txt command to tell them where your sitemap is.

Sitemap: http://www.mysite.com/sitemap.xml

*Bear in mind that not all bots behave and abide by the robots.txt file. They may not all be programmed to use the sitemap command.

[markvredeveld]

I have no control over the site I am linking to. All I want to do is stop a robot from following a couple of links on each of my webpages.

[SemAdvance]

You can do the nofollow via the links on the page.

Old way of outbound linking

Visit my <a href="http://www.example.com/">Smarter SEO</a> site.

Using a nofollow tag.

Visit my <a href="http://www.example.com/" rel="nofollow">Smarter SEO</a> site.

Peace!

[markvredeveld]

So if I put: rel="nofollow"> at the end of a link, a robot will not follow that link?

[chowell]

The robot may follow the link for discovery purposes, but will not pass any SEO or PageRank value.

[craigmn3]

One way to overcome this problem is to open up the other site on a frame page of your own. with a little banner at the top that Identifies you in a header frame then the other site in the body frame. Save all these links pages into one folder on your site and then use the robots.txt file to disallow the folder


going from :

My Site -> Your Site


to :

My Site -> My Frames Page -> Your Site

(edit)
and since you don't want these pages indexed the fact that they are in nasty frame pages won't matter

[Orion]

SE's are pretty good at indexing frames now though so not sure how well that would work.

the rel="nofollow" is a great tool for specific links within a page but not all SE's will listen to the command, Google does and that's what most people care about.

I find sometimes though that second tier engines may not and they may index the link then 1st tiers might pick up the links via third party engines / directories.

Robot.txt is the same not all 2nd tiers use them however it is the industry standard and MOST do respect them.

[wige]

If it is a paid link or just a matter of not wanting pagerank to be passed, you can also redirect to an interstatial 302 redirect page, which does not transfer pagerank.

[TrafficProducer]

You may find some help here:-

Robotstxt.org and Database of Web Robots, Overview