View Single Post
  #2 (permalink)  
Old 11-24-2008, 11:08 AM
Dubbya's Avatar
Dubbya Dubbya is offline
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Nov 2006
Location: Steinbach, Manitoba, Canada
Posts: 1,300
Dubbya RepRank 4Dubbya RepRank 4Dubbya RepRank 4Dubbya RepRank 4Dubbya RepRank 4
Default Re: Robot files? how important?

Essentially, a robots.txt file is a simple way of preventing pages, folders and images from getting indexed. Using one can save you a ton of bandwidth while allowing you to specify which pages appear in search engine results pages (SERPs). Generally a good idea, especially if you're running an ecommerce site.

About tobots.txt files, FAQ and more:

The Web Robots Pages

More info:
Inside Google Sitemaps: Using a robots.txt file

Sitemaps help ensure that your site gets indexed by making it easier for search engines to find linked pages. Again, this is particularly helpful when running an ecommerce site. You don't want to end up hiding content from potential customers simply because the SE's don't know about all of your pages.

Here's a pretty typical free online sitemap generator which should meet your needs:
Create your Google Sitemap Online - XML Sitemaps Generator

Selecting a "daily" change frequency won't likely offer any benefits unless you make changes to all of your files every day. Weekly or monthly will probably do just fine.

The "Last Modified" date is pretty much redundant as the spider will most likely get that information from the server anyway.

Save the finished sitemap.xml file to the root of your web and link to it in your robots.txt file using the following code:

Code:
Sitemap: http://www.typeyourURLhere.com/sitemap.xml
If you've already set up Google Analytics, Yahoo Sitemaps and MSN Webmaster Live accounts, you can now tell them to start using your sitemap file or just wait for them to find the sitemap link when the robots.txt file is automatically indexed.

.02
Reply With Quote