PLease Help With Robots.txt File

[savantcreative]

I am trying to create a robots.txt file for a Wordpress site I just launched but I don't know what to allow and what to disallow. I have had little luck learning about it in the Wordpress Codex and forum. Can someone please tell me what I need to do and perhaps paste a copy of one here that I can use? Thanks so much for your help.

[williamc]

From the codex page http://codex.wordpress.org/Search_Engine_Optimization_for_wordpress

Under the 'See also' section:

http://www.markwilson.co.uk/blog/2007/04/removing-duplicate-search-engine-content-using-robotstxt.htm

http://www.askapache.com/seo/seo-with-robotstxt.html

[savantcreative]

Would you use the one from the Codex verbatim? That's what I couldn't figure out. Must you use one? Thanks.

[williamc]

Every install and 'want' is going to be different per a persons needs. Use what you feel is necessary from them both, or use Joost devalks SEO plugin which takes care of that for you.

[savantcreative]

Thanks. Can you please check out http://www.nylegalblog.com/robots.txt and tell me what is going on here?

[williamc]

It looks like wordpress is not seeing the robots file and automatically redirecting to the homepage. You may need to have wp check for the existance of .txt file as well as php|html in the rewrites.

[savantcreative]

This is not my blog. I think it is deliberate.

[savantcreative]

It's not redirecting to the home page because, although it is the hp, it has a different url

[williamc]

True it is not 'redirecting' that was a bad choice of wording on my part, but it is in the rewrites either way. The rewrite is telling it which pages content to use, but it is not changing the url as a 301 or 302 rule would do. It still amounts to the same thing as I said above.

[savantcreative]

Do you know how to make this happen just as in the example?

[williamc]

Do you know how to make this happen just as in the example?

Clarify please. How to make 'what' happen exactly?

[savantcreative]

I would like to have the url http://example.com/robots.txt display the home page rewritten as http://example/robots.txt. In other words, I do not want to show a robots.txt file. Thanks

[williamc]

I would like to have the url http://example.com/robots.txt display the home page rewritten as http://example/robots.txt. In other words, I do not want to show a robots.txt file. Thanks

ok.



RewriteEngine on
RewriteRule ^robots.txt /index.html


or index.php, whatever.

One question however. Why not simply make a blank file and name it robots.txt?

[savantcreative]

I want the resulting page url to be http://example.com/robots.txt but be a copy of the http://example.com page. Is that what this will do? If so, where do I add it? I already have a ,htaccess file. Do I just ass this to it?
In answer to your question, I don't want a black robots.txt page. It is an experiment I am conducting.
Thanks :)

[williamc]

You would add all or part of it to your existing .htaccess file depending on whats in there already. Placement in the file does also matter. If you would like to paste in PM, your .htaccess file I will be happy to show you where to put the rewrite in it.

[savantcreative]

I just sent it. You're the best!

[williamc]

Replied with instructions on how to edit your specific .htaccess

[savantcreative]

I just sent you a new PM

[williamc]

When using rewrites to aim visitors to a given place for a certain file, the file must not exist on the server. Remove the existing robots.txt file and the rewrite will handle it properly.

[savantcreative]

I can't find a separate robots.txt file. When I remove the .htaccess file I get a 404 for robots.txt so it seems to be a part of it

[williamc]

The yes, it may well be built into WP in this case. However I had you place the rewrite above the WP stuff so it should have triggered before any of that. Are you sure you pasted me the entire .htaccess? Or did you just paste me the WP stuff?

[savantcreative]

I pasted the entire file

[savantcreative]

The weird thing is that when I remove the .htaccess file I get a 404 error for the robots.txt file. I spoke with the hosting company and they said this can't be done.

[williamc]

Ok, in that case I am going to give you new contents for the .htaccess file in PM. I suggest you keep the original somewhere safe so you can reupload if needed (as in this new one breaks anything).

[savantcreative]

Great Thanks!

[savantcreative]

Now I get a 4040 error for the page