Robot files? how important?

[The Lady]

I have a robot file, on one site... I'm trying to get a feel for what I need and how to do it right. How important are they? Also, considering a sitemap - no clue how to create one, could some one fill in the otherwise website illiterate;) person please....

i can market - i can write - and can persuade but can't get these things to look the way i want them too... ;)

[Dubbya]

Essentially, a robots.txt file is a simple way of preventing pages, folders and images from getting indexed. Using one can save you a ton of bandwidth while allowing you to specify which pages appear in search engine results pages (SERPs). Generally a good idea, especially if you're running an ecommerce site.

About tobots.txt files, FAQ and more:

The Web Robots Pages (http://www.robotstxt.org/)

More info:
Inside Google Sitemaps: Using a robots.txt file (http://sitemaps.blogspot.com/2006/02/using-robotstxt-file.html)

Sitemaps help ensure that your site gets indexed by making it easier for search engines to find linked pages. Again, this is particularly helpful when running an ecommerce site. You don't want to end up hiding content from potential customers simply because the SE's don't know about all of your pages.

Here's a pretty typical free online sitemap generator which should meet your needs:
Create your Google Sitemap Online - XML Sitemaps Generator (http://www.xml-sitemaps.com/)

Selecting a "daily" change frequency won't likely offer any benefits unless you make changes to all of your files every day. Weekly or monthly will probably do just fine.

The "Last Modified" date is pretty much redundant as the spider will most likely get that information from the server anyway.

Save the finished sitemap.xml file to the root of your web and link to it in your robots.txt file using the following code:



Sitemap: http://www.typeyourURLhere.com/sitemap.xml


If you've already set up Google Analytics, Yahoo Sitemaps and MSN Webmaster Live accounts, you can now tell them to start using your sitemap file or just wait for them to find the sitemap link when the robots.txt file is automatically indexed.

.02

[The Lady]

thank you!

[ldylion214]

Sitemap: http://www.typeyourURLhere.com/sitemap.xml


.02

I linked this in my robot.txt file, http://www.tribeazure.com/urlist.txt. I opted to download the text file. Does this make a difference and did I do this correctly?

Thanks! Nicci

[Dubbya]

Nicci, you no longer need urllist.txt files for Yahoo!. They're only too happy to find a sitemap.xml file. Just make sure you link to it in your robots.txt file.

Besides, using a single xml file for Google, Yahoo! and MSN will save you a little work in the long run.

[ldylion214]

THank you Dubbya. I changed it.

Have a great Thanksgiving! Nicci