I'm optimizing a site that was built in coldfusion. Urls are _reasonably_ search engine friendly but I need to block a few pages using robots.txt.
Urls on the site appear like this in the address bar, with no extension:
http://www.domain.com/go/our-products/name-of-the-page
My question is, does robots.txt need a file extension? Is this a totally noob question?
Would this work in robots.txt?
User-Agent: *
Disallow: /go/our-products/name-of-the-page
TIA for any advice.