robots.txt question

braknews · #1 (**permalink**) 05-10-2005, 12:06 PM

I have never seen a '?' in a robots.txt before.

Does the code below mean that no bot will spider the site?

Code:

User-agent: *
Disallow: /*?
Disallow: /?

Thanks

jestep · #2 (**permalink**) 05-10-2005, 12:45 PM

My guess is that they are trying to block all dynamic pages.

I'm not sure if it works or not, but it would be a good way to keep a robot from crawling link pages if they are dynamically generated. They would still appear to have Page Rank even though they aren't spidered. Google subtracts 1 PR from a dynamic page with the same name as its root.

IE:
info.php - PR5
info.php?page=this one - PR4

The dynamic page would remain unspidered with Page Rank.

timmathews.com · #3 (**permalink**) 05-11-2005, 04:12 PM

I agree 100%
This code in the robots.txt is trying to block spiders from crawling the dynamic pages.