just wondering, is robot.txt a must or would meta tags will do the job? by the way i don't want to disallow any content on the site!
No, you don't have to have a robots.txt file. But I suggest having one as you can keep a lot of rogue robots off of your site by using it.
There are a lot of spiders out there that will try to crawl your site to get email addresses and links and other things not related to search engine rankings.
Why would a rogue robot obey robots.txt?
I think it's worth uploading a blank robots.txt just to cut out the 404s.
Also it has occurred to my in the past that if you have a custom 404 page and no robots.txt, SEs may think that your robots.txt is spammie. I know that this is only a slim chance as the server should still return a 404 error code, however a server config error or a robot error could result in you getting branded a spammer. I know this is very unlikley, but I would rather cut out the possibility all the same!
I am new to this web site but I have to admit it looks great and there so much content here
I want to get involve in seo but before I do that I want to know
When you create your meta tag in your html pages and add a tag such as the one below
<meta name="language" content="English" />
<meta HTTP-EQUIV="expires" CONTENT="0">
<meta name="author" content="Direct2net" />
<meta name="copyright" content="Direct2net" />
<meta name="robots" content="index,follow">
<meta name="revisit-after" content="1 days" />
Doesn’t that do the same to Google spider and get it spied the site
Do I have to have a file on my server called robot.txt ?
I hope you guys to see me as fool (if you do hide your face)
If there is a sample of such file (robot.txt) can someone provide me with a link that contain all the options you need in order to create it
Wow!!! what a great response
That should give me agood start