iEntry 10th Anniversary Forum Rules Search
WebProWorld
Register FAQ Calendar Mark Forums Read
Search Engine Optimization Forum SEO is much easier with help from peers and experts! The WebProWorld SEO forum is for the discussion and exploration of various search engine optimization topics. Any non (engine) specific SEO or SEM topics should go here.

Share Thread: & Tags

Share Thread:

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 05-28-2008, 04:42 AM
WebProWorld Veteran
 
Join Date: Jun 2004
Location: Australia
Posts: 531
watto RepRank 2watto RepRank 2
Default robots.txt file Question

I recently made some changes to my robots.txt file and I would like some feedback as to whether this is correct:

http://www.business-trader.com.au/robots.txt

All help is greatly appreciated.

watto
Reply With Quote
  #2 (permalink)  
Old 05-28-2008, 05:00 AM
caravan's Avatar
WebProWorld Pro
 
Join Date: May 2006
Location: Preston, Lancashire, UK
Posts: 103
caravan RepRank 1
Default Re: robots.txt file Question

Hi Watto,

Have you tried running it through the Google Webmaster Analyse robots.txt Tool. You simply put in the contents of your robots file and a list of your websites url's and Google will show you how the file affects your website pages. It is an excellent way of debugging the file.
Reply With Quote
  #3 (permalink)  
Old 05-28-2008, 05:03 AM
WebProWorld Veteran
 
Join Date: Jun 2004
Location: Australia
Posts: 531
watto RepRank 2watto RepRank 2
Default Re: robots.txt file Question

Thanks for the tip!

Regards

watto
Reply With Quote
  #4 (permalink)  
Old 05-30-2008, 02:06 AM
incrediblehelp's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Jan 2004
Location: Live in Cincy Now
Posts: 7,573
incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4
Default Re: robots.txt file Question

Watto their is no right or wrong for robots.txt. Just whatever works for your website.
Reply With Quote
  #5 (permalink)  
Old 05-30-2008, 06:18 AM
WebProWorld Veteran
 
Join Date: Jun 2004
Location: Australia
Posts: 531
watto RepRank 2watto RepRank 2
Default Re: robots.txt file Question

Thanks incrediblehelp. I did realise that robots.txt files are different and a robots.txt file needs to be customized to your site, I really just wanted some feedback to make sure I had used correct coding.

Cheers

watto
Reply With Quote
  #6 (permalink)  
Old 05-30-2008, 08:35 AM
incrediblehelp's Avatar
WebProWorld 1,000+ Club
WebProWorld MVP
 
Join Date: Jan 2004
Location: Live in Cincy Now
Posts: 7,573
incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4incrediblehelp RepRank 4
Default Re: robots.txt file Question

I would also add:

Disallow: /cgi-bin
Disallow: /_private
Disallow: /_vti_bin
Disallow: /_vti_cnf
Disallow: /_vti_log
Disallow: /_vti_pvt
Disallow: /_vti_txt
Reply With Quote
  #7 (permalink)  
Old 05-30-2008, 09:20 AM
WebProWorld Veteran
 
Join Date: Jun 2004
Location: Australia
Posts: 531
watto RepRank 2watto RepRank 2
Default Re: robots.txt file Question

Thanks for the tip!

Kind Regards

watto
Reply With Quote
Reply

  WebProWorld > Search Engines > Search Engine Optimization Forum

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
What do I need to know about the robots.txt file? emersonworldwide Google Discussion Forum 13 09-21-2007 07:57 AM
robots.txt file question kruser IT Discussion Forum 2 09-13-2007 06:15 PM
PR 6 for Robots.txt file amar Search Engine Optimization Forum 2 12-26-2006 09:57 PM
Robots .txt File Xen2 Search Engine Optimization Forum 7 11-14-2006 05:19 AM
What is a robots.txt file? Tamelyne Search Engine Optimization Forum 3 10-25-2004 08:42 PM


All times are GMT -4. The time now is 10:58 PM.



Search Engine Optimization by vBSEO 3.3.0