View Single Post
  #9 (permalink)  
Old 09-07-2007, 10:52 AM
wige's Avatar
wige wige is online now
Moderator
WebProWorld Moderator
 

Join Date: Jun 2006
Location: United States
Posts: 1,843
wige RepRank 4wige RepRank 4wige RepRank 4wige RepRank 4
Default Re: Sitemap used as a replacement for robots.txt

One of the things with redirects is the message they send the spider when the redirecting page is indexed. When a 302 redirect is encountered, the search engine will continue to check the referring page to see if the content is no longer being redirected. If you specify a 301 redirect, the spider will automatically calculate links to the old location as being links to the new location, and rechecks of the redirecting page are greatly reduced.

Probably the primary consideration is how often you expect to change the destination of the redirects. If they will never, or almost never, change, use a 301. As far as technical issues, bots usually look at urls containing paramaters as seperate pages, so in theory these should not cause crawl errors unless the redirect is not in a proper format, and in these cases the URL would be listed under not crawled rather than blocked by robots.txt.
__________________
The best way to learn anything, is to question everything.
Reply With Quote