View Single Post
  #10 (permalink)  
Old 01-18-2008, 03:20 PM
Webnauts's Avatar
Webnauts Webnauts is offline
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Aug 2003
Location: Worldwide
Posts: 8,132
Webnauts RepRank 8Webnauts RepRank 8Webnauts RepRank 8Webnauts RepRank 8Webnauts RepRank 8Webnauts RepRank 8Webnauts RepRank 8Webnauts RepRank 8Webnauts RepRank 8
Default Re: Opinion? Would changing your Wordpress theme have an effect on optimization?

Quote:
Originally Posted by mjtaylor View Post
What would you recommend on editing robots.txt? I have been told to disallow archives ... do you agree? What else?
There are several folders/pages that should be disallowed. I just created for our members here a robots.txt that can be implemented right away for Wordpress without problems. And it will help you to get rid of most, if not all of pages found in Googles the supplemental results, which cause a harmful leak of PR.

User-agent: Googlebot
Disallow: /*?*
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.tar$
Disallow: /*.tgz$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /category/*/*
Disallow: */comments
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/*/feed/$
Disallow: /*/*/*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/*/feed/rss/$
Disallow: */trackback
Disallow: /*/trackback/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/trackback/$
Allow: /wp-content/uploads

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-register.php
Disallow: /wp-login.php
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
Disallow: /search
Disallow: /stats/
Disallow: /contact/
Disallow: /about/
Disallow: /archives/
Disallow: /dh_

There are probably more folders or files that might need to be disallowed. For example as you mentioned, the archives.

I hope this is useful for you and others here.
__________________
"Being an expert isn't telling other people what you know. It's understanding what questions to ask, and flexibly applying your knowledge to the specific situation at hand. Being an expert means providing sensible, highly contextual direction." Jeff Atwood
SEO Workers - Search Engine Optimization Consulting Company | SEO Analysis Tool | Webnauts Net SEO

Last edited by Webnauts; 01-18-2008 at 05:18 PM.
Reply With Quote