How to prevent bots from crawling individual links?

[Ozzman]

My scenarios is that bots are already disallowed to visit the things they are not supposed to have access to. However, the problem is that there may be LINKS to such threads (for example, links to private threads in Forum's "recent posts" section) or, what is more interesting, links to various 'Reply', 'Add Post' and similar buttons.

Search bots see such buttons and attempt to navigate to the related links, which results in the errors similar to those that we can see.

I don't really like to provide different content (i.e. some sort of conditional code that would remove such buttons for search bots), especially taking into account that some software might set custom or empty user agents, which will look similar to some bots..in such cases it might be a user who wants to ensure his privacy and hide browser information and we need to ensure that such users are able to view all those 'add post' buttons.

Taking it into account, i wanted to check if there is some good in terms of SEO way to specify that this or that link (meaning links to private threads or buttons that would bring user to 'add post' page) is not meant to be searched.

Please share your thoughts on this.

[mjtaylor]

It seems to me that you could use the nofollow attribute ...

[Ozzman]

Originally Posted by mjtaylor

It seems to me that you could use the nofollow attribute ...

I would include noindex,follow version of META tag in such pages...however, the bots get redirected to 304 BEFORE getting to those inaccessible pages (thus increasing our 304 statistics), that's why i asked about the way (if such way exists) to specify that EXACT links should not be FOLLOWED by bots.

[PAUL-B]

For crawlers, Meta tag attributes (nofollow) & Robot.txt are both there to control their activity. But something to specify that EXACT links... Can't think of anything. Let me know if you do find something.

[incrediblehelp]

Have you seen this article?

A Loophole For Paid Links | WebProNews

[Webnauts]

You can probably use a server side referrer script.

[kgun]

Depending on how selective you want to be, here are my options:

1. Exclude bad and good bots from (part of) your site placing a .htaccess file in relevant folders. That is most secure since it is on the lowest level.
2. Exclude the robots from visiting part of your site like WPW has done it in their robots.txt file
3. Put nofollow on meta tags for the relevant pages.
4. Pute rel="nofollow" on individual links.
5. Buy this The*Art*& Science*of JavaScript - SitePoint Books excellent book and read chapter 2, especially the sections with heading "Server-side Badgets" and "Custom Client-side Badgets" respectively.

[tmaster]

You should include a robots.txt file to tell the good bots not to visit your reply links.

Only problem is that only the good bots will follow the rules you set.

Suggest you also install Bad Behavior. http://www.ioerror.us/software/bad-behavior

It can be installed on any PHP script. Doesn't need the WordPress database that it uses on WP

[torpengkute]

You can disallow crawling your individual page through on your robots.txt file.

[Webnauts]

Originally Posted by PAUL-B

For crawlers, Meta tag attributes (nofollow) & Robot.txt are both there to control their activity. But something to specify that EXACT links... Can't think of anything. Let me know if you do find something.

Maybe as an alternative server side scripting referrers?

[torpengkute]

Originally Posted by Webnauts

Maybe as an alternative server side scripting referrers?

maybe you can configure on your .htaccess File.

[Webnauts]

Originally Posted by torpengkute

maybe you can configure on your .htaccess File.

Hey that sounds cool. I usually do that with PHP. Can you explain how can that work?

[torpengkute]

Your htaccess file is a ticket to solving a lot of your search engine problems. Not all of them, but some of them. It’s a file on your server that gives instructions to browsers and search engine spiders, telling them how to read your web pages. One common usage of this file is to use it to redirect old web pages to new web pages. Frequently, webmasters will update their information and when doing so will change the URL of a web page. Well, if you do that then you still have that old web page indexed and when people try to visit that page they will get a 404 error page. To prevent that from happening, you can add a 301 redirect command in your htaccess to redirect traffic to your new page.

[Webnauts]

Originally Posted by torpengkute

Your htaccess file is a ticket to solving a lot of your search engine problems. Not all of them, but some of them. It’s a file on your server that gives instructions to browsers and search engine spiders, telling them how to read your web pages. One common usage of this file is to use it to redirect old web pages to new web pages. Frequently, webmasters will update their information and when doing so will change the URL of a web page. Well, if you do that then you still have that old web page indexed and when people try to visit that page they will get a 404 error page. To prevent that from happening, you can add a 301 redirect command in your htaccess to redirect traffic to your new page.

You are going far off-topic.

The question once again: How can someone disallow crawling an individual page with the .htaccess file?

[kgun]

Originally Posted by Webnauts

Hey that sounds cool. I usually do that with PHP. Can you explain how can that work?

I am on a shared hoster for some of my sites, and do not have access to php.ini and other configuration files, so I have to do it in .htaccess and it is more secure, since it is closer to the web server. So if you can do something in a php configuration file and the same task can be done in a web server configuration file, you should prefer the last. It is simply more secure.

Originally Posted by Webnauts

You are going far off-topic.

The question once again: How can someone disallow crawling an individual page with the .htaccess file?

If you have one page in each folder, like I have on my largest site. See the second link in my signature. place a .htaccess file in that folder and disallow bots, referrers or what you prefer. You have full control. It is more secure.

Take your time, read, reread and enjoy your increased security:
Network and System Administration I 10

[Webnauts]

Kjell, thanks for the valuable information, but be aware that what you are saying is nothing new to me.

Our member Torpengkute said that we can disallow crawling pages with .htaccess, and I am asking him or anybody else here, how can that work!!!

I use a rule in my .htaccess files, to make sure that my robots.txt file will not get indexed and won't show up in the search results, like happens here http://www.whitehouse.gov/robots.txt
And that because I do not want to experience a PR dilution as they do, since their robots.txt has a PR 4.
And please don't start now with, John they do not need to care. It is just an example.

Hey, this is my 6,000th post here at WebProWorld.

[ctabuk]

Cool John - if I delete say, 5,999 you can start all over again!1

Shall I do it boys and girls??





Congratulations John

[nipplecharms1]

6,000 posts. John, get outside and smell the flowers!

M

[kgun]

Originally Posted by Webnauts

Kjell, thanks for the valuable information, but be aware that what you are saying is nothing new to me.

Our member Torpengkute said that we can disallow crawling pages with .htaccess, and I am asking him or anybody else here, how can that work!!!

I use a rule in my .htaccess files, to make sure that my robots.txt file will not get indexed and won't show up in the search results, like happens here http://www.whitehouse.gov/robots.txt
And that because I do not want to experience a PR dilution as they do, since their robots.txt has a PR 4.
And please don't start now with, John they do not need to care. It is just an example.

1. John congratulation with your post number 6000.
2. Nothing new to you. It should not be: An interesting test.
3. Note: By blocking access to your page(s), doesn't that imply that the bot does not crawl the page? If you want the bot to index the page, that is of course not an option.
4. Reread my first post in this thread and the order of priority. I am not sure that you have understood the referrer method good enough either. That was the reason for my 5th point in that thread.

Related thread:
Updated .htaccess to Block spammers / scammers.

You find related threads by the following WPW site search:

1. Click the search button beneath the "Subscribe" button.
2. Click "Advanced Search" below the "Search Forums" button (Not below "Search Tags").
3. Put kgun in the User name field.
4. Choose "Find Threads Started by User"
5. Choose "Internet Security Discussion Forum" in "Search Forum(s)"
6. Put .htaccess in the "Key Word(s)" field in "Search by Key Word" at the left of the page.
7. Click "Search now"

Finally:

1. How to prevent bots from crawling individual links?
2. Assumption: Crawling all links on a page or part of the site and not allowing the bots to index and / or archieve pages.

Example:

1. http://www.forumnorway.com/
2. http://kjellbleivik.com/Books/
   

Do you or any other referrer or any Bot have access? If you have, I have done something wrong. The last example has a .htaccess file in the Books subfolder. You will have access to my home bage.

So to conclude:
Start by using server side configuration files like .htaccess. Then go on to robots.txt to be more specific, then meta tags...

And it has one great advantage: It saves bandwith and burden on your web server if you can do it in web sever configuration files. Finally it is more secure.

[Webnauts]

OK Kgun. I think you did not get my point.

The user was claiming that we can use .htaccess rules to disallow crawlers indexing and following pages.

Individual pages should be accessed by users, but not from certain crawlers, like i.e Googlebot.

Looks like he gave up, as I assume he could not provide us here with any examples.

To be honest I use several server side scripting or configuration methods to achieve all above, but I want to hear how others do that. Got it know?

So can you please place an example of code here and stop referring to other threads, etc?

Thanks bro.

[kgun]

Point taken.

Save bandwith and choose the most secure solution where you have an option.

[Webnauts]

Originally Posted by kgun

Point taken.

Save bandwith and choose the most secure solution where you have an option.

I am sure that when it comes to SE,

If-Modified-Since = "If-Modified-Since" ":" HTTP-date

is the most appropriate solution.

By the way, do you think this could disallow crawlers to index a page?

<FilesMatch "example\.html">
Header set X-Robots-Tag "noindex"
</FilesMatch>

I think I have to test that myself anyway. But just asking...

[torpengkute]

Originally Posted by Webnauts

OK Kgun. I think you did not get my point.

The user was claiming that we can use .htaccess rules to disallow crawlers indexing and following pages.

Individual pages should be accessed by users, but not from certain crawlers, like i.e Googlebot.

Looks like he gave up, as I assume he could not provide us here with any examples.

To be honest I use several server side scripting or configuration methods to achieve all above, but I want to hear how others do that. Got it know?

So can you please place an example of code here and stop referring to other threads, etc?

Thanks bro.

Hey bro!,

As you said " To be honest I use several server side scripting or configuration methods to achieve all above" So why do you want to hear other suggestions...

So It means you did not really know that... Sorry john, before i want to help you how to do that But......

anyway good luck to you bro!

[kgun]

Originally Posted by Webnauts

I am sure that when it comes to SE,

If-Modified-Since = "If-Modified-Since" ":" HTTP-date

is the most appropriate solution.

By the way, do you think this could disallow crawlers to index a page?

<FilesMatch "example\.html">
Header set X-Robots-Tag "noindex"
</FilesMatch>

I think I have to test that myself anyway. But just asking...

That seems like a .htaccess statement, I have not seen, so I think you have to test it yourself. I am far from an .htaccess expert, but since I have written about the security layer as the first layer of the web, I should have known more.

I am sure, if someone (you) became an Apache server expert, you will soon laugh at the spammers, hackers and intruders.

Related links:

Apache Tutorial: .htaccess files

Documentation: Apache HTTP Server - The Apache HTTP Server Project

30 minutes training each day, and you can offer network security services in addition to SEO services.

And John, you should set up your own paid extranet.

[Webnauts]

Originally Posted by torpengkute

Hey bro!,

As you said " To be honest I use several server side scripting or configuration methods to achieve all above" So why do you want to hear other suggestions...

So It means you did not really know that... Sorry john, before i want to help you how to do that But......

anyway good luck to you bro!

Excuses? Forget it.

You wanted to help? Did you all here hear that?
(Sorry for my arrogance, but that was honestly too funny.)

If you follow the link to the other thread Kgun posted, you will see that I am not telling stories.

And the mods and other older members here can confirm that I am not a guy telling stories here but facts!

So if it is true what you are claiming, show your facts or just let give up theories, as they do not help anybody here.

OK?

[kgun]

Originally Posted by Webnauts

So if it is true what you are claiming, show your facts or just let give up theories, as they do not help anybody here.

Ah John. Have you not noted my signature at David's forum?

"If the fact's don't fit the theory, change the facts."