|
|
||||||
|
||||||
| Index Link To US Private Messages Archive FAQ RSS | ||||||
| Search Engine Optimization Forum SEO is much easier with help from peers and experts! The WebProWorld SEO forum is for the discussion and exploration of various search engine optimization topics. Any non (engine) specific SEO or SEM topics should go here. |
Share Thread: & Tags
|
||||
|
![]() |
|
|
LinkBack | Thread Tools | Display Modes |
|
|||
|
I have never seen a '?' in a robots.txt before.
Does the code below mean that no bot will spider the site? Code:
User-agent: * Disallow: /*? Disallow: /?
__________________
Subbuteo in Los Angeles, California |
|
|||
|
My guess is that they are trying to block all dynamic pages.
I'm not sure if it works or not, but it would be a good way to keep a robot from crawling link pages if they are dynamically generated. They would still appear to have Page Rank even though they aren't spidered. Google subtracts 1 PR from a dynamic page with the same name as its root. IE: info.php - PR5 info.php?page=this one - PR4 The dynamic page would remain unspidered with Page Rank. |
![]() |
|
| Thread Tools | |
| Display Modes | |
|
|
|
WebProWorld |
Advertise |
Contact Us |
About |
Forum Rules |
MVP's |
Archive |
Newsletter Archive |
Top |
WebProNews
WebProWorld is an iEntry, Inc. ® site - © 2009 All Rights Reserved Privacy Policy and Legal iEntry, Inc. 2549 Richmond Rd. Lexington KY, 40509 |