iEntry 10th Anniversary Forum Rules Search
WebProWorld
Register FAQ Calendar Mark Forums Read
Search Engine Optimization Forum SEO is much easier with help from peers and experts! The WebProWorld SEO forum is for the discussion and exploration of various search engine optimization topics. Any non (engine) specific SEO or SEM topics should go here.

Share Thread: & Tags

Share Thread:

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 01-12-2009, 11:13 PM
Web-Design-Guy's Avatar
WebProWorld Pro
 
Join Date: Jun 2008
Location: Perth, Western Australia
Posts: 110
Web-Design-Guy RepRank 1
Default robots.txt question / coldfusion

I'm optimizing a site that was built in coldfusion. Urls are _reasonably_ search engine friendly but I need to block a few pages using robots.txt.

Urls on the site appear like this in the address bar, with no extension:

http://www.domain.com/go/our-products/name-of-the-page

My question is, does robots.txt need a file extension? Is this a totally noob question?

Would this work in robots.txt?

User-Agent: *
Disallow: /go/our-products/name-of-the-page

TIA for any advice.
Reply With Quote
  #2 (permalink)  
Old 01-13-2009, 10:35 AM
wige's Avatar
Moderator
WebProWorld Moderator
 
Join Date: Jun 2006
Location: United States
Posts: 2,648
wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9wige RepRank 9
Default Re: robots.txt question / coldfusion

Extensions are not required. Exclusions in robots.txt are processed as text strings from left-to-right, so you only need to enter the string as far as needed to be unique - so if you wanted to block every page starting with name-of-, you can do that by just leaving off the last word.
__________________
The best way to learn anything, is to question everything.
Reply With Quote
  #3 (permalink)  
Old 01-13-2009, 02:24 PM
WebProWorld New Member
 
Join Date: Jan 2009
Posts: 7
Markos RepRank 0
Default Re: robots.txt question / coldfusion

You can test your robot.txt file using google webmaster tools.
Reply With Quote
Reply

  WebProWorld > Search Engines > Search Engine Optimization Forum

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Robots.txt Question watto Search Engine Optimization Forum 33 12-11-2008 01:09 PM
robots.txt question kimber23 Search Engine Optimization Forum 4 12-05-2006 05:51 PM
robots.txt question braknews Search Engine Optimization Forum 2 05-11-2005 04:12 PM
Robots.txt question dyno Google Discussion Forum 2 08-24-2004 11:13 PM
A Robots.txt question braknews Google Discussion Forum 3 08-20-2004 03:31 PM


All times are GMT -4. The time now is 02:19 AM.



Search Engine Optimization by vBSEO 3.3.0