Best Robots.txt wordpress

[innominds]

Recently I've got a wordpress blog at www.domain.com/blog/
With the recent changes in Google algorithm, I would like to take concrete steps in making it more SEO friendly by removing the low quality pages with tags.
I mean to say I do want to remove the indexing of tag and other low quality ages that decrease the value of main blog posts.
For this I need to write a good robots.txt
Could anyone help me in getting the best robots.txt for my wordpress blog placed at www.domain.com/blog/

Thanks in advance

[mjtaylor]

There is a basic instruction page and automated tool from Google here: http://www.google.com/support/webmas...n&answer=40360.

[innominds]

Hey! Thanks for the reference. I've gone through it.

After research I've found this robots.txt:

====================

User-agent: Googlebot
Disallow: /wp-content/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /feed/
Disallow: /archives/
Disallow: /sitemap.xml
Disallow: /index.php
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: */trackback/
Disallow: /page/
Disallow: /tag/
Disallow: /category/

User-agent: Googlebot-Image
Disallow: /wp-includes/

User-agent: Mediapartners-Google*
Disallow:

User-agent: ia_archiver
Disallow: /

User-agent: duggmirror
Disallow: /


========================

As I said above, I'm installing the blog at www.domain.com/blog/ , so do I need to change the above robots.txt file?


One more query! I want to block the robots from accessing a particular web page www.domain.com/XYXABC.html then how to include that in the txt file?

[AboutWeb]

LoL! Disallow: /sitemap.xml
Why would you disallow your sitemap ? The sitemap.xml file purpose is to tell search engines about your pages and you block Google ?

[deepsand]

And, right below that we see

Disallow: /index.php

[innominds]

I'm sorry! Is this OK now?

User-agent: Googlebot
Disallow: /wp-content/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /feed/
Disallow: /archives/
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: */trackback/
Disallow: /page/
Disallow: /tag/
Disallow: /category/

User-agent: Googlebot-Image
Disallow: /wp-includes/

User-agent: Mediapartners-Google*
Disallow:

User-agent: ia_archiver
Disallow: /

User-agent: duggmirror
Disallow: /

[deepsand]

In the first instance, why block only Google?

And, you want to block crawling of all pages with the PHP extension?

[AboutWeb]

I agree with deepsand that you must use the * for all search engine spiders.

For the second, on a wordpress site all articles/pages doesn't have a php extension. So he would block the wordpress core files by blocking *.php .

You don't need to disallow /archives/ , just set them to nofollow by using All in One Seo Pack.
/category/ and /page/ is a bad idea to block them: I got Page Rank 2 on most of the pages and categories.

[deepsand]

He's got other PHP pages on his site.

[innominds]

Sorry for the mistakes!
Actually I'm poor at coding (php)
If you don't mind could you give me the best robots.txt for my wordpress blog at www.domain.com/blog/