do i have to have a robot.txt?

[saghir69]

just wondering, is robot.txt a must or would meta tags will do the job? by the way i don't want to disallow any content on the site!

[bhartzer]

No, you don't have to have a robots.txt file. But I suggest having one as you can keep a lot of rogue robots off of your site by using it.

There are a lot of spiders out there that will try to crawl your site to get email addresses and links and other things not related to search engine rankings.

[HardCoded]

Why would a rogue robot obey robots.txt?

[bhartzer]

It's odd, but some of them do.

[Martin Smith]

I think it's worth uploading a blank robots.txt just to cut out the 404s.

Also it has occurred to my in the past that if you have a custom 404 page and no robots.txt, SEs may think that your robots.txt is spammie. I know that this is only a slim chance as the server should still return a 404 error code, however a server config error or a robot error could result in you getting branded a spammer. I know this is very unlikley, but I would rather cut out the possibility all the same!

[direct2net]

Hi There
I am new to this web site but I have to admit it looks great and there so much content here

I want to get involve in seo but before I do that I want to know
When you create your meta tag in your html pages and add a tag such as the one below
<meta name="language" content="English" />
<meta HTTP-EQUIV="expires" CONTENT="0">
<meta name="author" content="Direct2net" />
<meta name="copyright" content="Direct2net" />
<meta name="robots" content="index,follow">
<meta name="revisit-after" content="1 days" />

Doesn’t that do the same to Google spider and get it spied the site

Do I have to have a file on my server called robot.txt ?

I hope you guys to see me as fool (if you do hide your face)

If there is a sample of such file (robot.txt) can someone provide me with a link that contain all the options you need in order to create it

[cbp]

Welcome to WPW.

Here is a good thread to read on robots.txt:
http://www.webproworld.com/viewtopic.php?t=13110

CBP

[Dave Hawley]

IMO, you only need a Robot text file to stop robots. In other words, most don't need them.

[Mel]

Unless of course you get tired of your server putting up a 404 error every time a bot visits.

[direct2net]

Wow!!! what a great response

That should give me agood start

Thank You