iEntry 10th Anniversary Forum Rules Search
WebProWorld
Register FAQ Calendar Mark Forums Read
Search Engine Optimization Forum SEO is much easier with help from peers and experts! The WebProWorld SEO forum is for the discussion and exploration of various search engine optimization topics. Any non (engine) specific SEO or SEM topics should go here.

Share Thread: & Tags

Share Thread:

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 12-04-2006, 07:58 PM
WebProWorld Member
 
Join Date: Oct 2006
Location: Dallas. TX
Posts: 30
kimber23 RepRank 0
Default robots.txt question

what purpose does it serve exactly? my understanding is that you can use it to tell the robots to not crawl certain pages.
if i don't need to tell the bots to stay away from anything on a site, do i need a robots.txt file for any other purpose?
Reply With Quote
  #2 (permalink)  
Old 12-04-2006, 09:37 PM
WebProWorld Pro
 
Join Date: Oct 2006
Posts: 102
andyf RepRank 0
Default

The main use of robot.txt is to tell different search engine spiders not to crawl or index certain sections or pages of your site.

Also you have choice to specify which robots should not get crawl through your site.
Some robots crawls your site very fast and many times which can affect your site’s performance, so using robot.txt file you can keep away such spiders which hurts you.
Reply With Quote
  #3 (permalink)  
Old 12-05-2006, 03:35 PM
incrediblehelp's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Jan 2004
Location: Live in Cincy Now
Posts: 7,573
incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4
Default

Well every major search engine looks for one when they visit your site. It seems they want to find one. I would add one just to avoid 404 errors, regardless of the other benefits of blocking "bad" spiders and directories.
Reply With Quote
  #4 (permalink)  
Old 12-05-2006, 05:41 PM
WebProWorld Member
 
Join Date: Oct 2006
Location: Dallas. TX
Posts: 30
kimber23 RepRank 0
Default

so, if i'm not trying to tell any bots not to come, what to i include in the file?
Reply With Quote
  #5 (permalink)  
Old 12-05-2006, 05:51 PM
incrediblehelp's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Jan 2004
Location: Live in Cincy Now
Posts: 7,573
incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4
Default

This example allows all robots to visit all files because the wildcard "*" specifies all robots:

Code:
User-agent: *
Disallow:
This example keeps all robots out:

Code:
User-agent: *
Disallow: /
Reply With Quote
Reply

  WebProWorld > Search Engines > Search Engine Optimization Forum

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 09:39 PM.



Search Engine Optimization by vBSEO 3.3.0