iEntry 10th Anniversary Forum Rules Search
WebProWorld
Register FAQ Calendar Mark Forums Read
Go Back   WebProWorld > Search Engines > Search Engine Optimization Forum
User-agent: *
Index Link To US Private Messages Archive FAQ RSS
Search Engine Optimization Forum SEO is much easier with help from peers and experts! The WebProWorld SEO forum is for the discussion and exploration of various search engine optimization topics. Any non (engine) specific SEO or SEM topics should go here.

Share Thread: & Tags

Share Thread:

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 02-01-2008, 08:47 PM
WebProWorld Member
  
Join Date: Oct 2004
Location: California
Posts: 42
benc007 RepRank 0
Default User-agent: *
In robots.txt, I see the following. Does this mean that all search engines should not crawl or index the site???



User-agent: *
Disallow:
Reply With Quote
  #2 (permalink)  
Old 02-02-2008, 01:44 AM
bobitza's Avatar
WebProWorld Veteran
  
Join Date: Jan 2004
Location: Canada
Posts: 303
bobitza RepRank 2bobitza RepRank 2
Default Re: User-agent: *
* = ALL users agents

... should

Disallow: = nothing (there is nothing after dissalow)

==> all search engines are allowed to crawl the entire site.
__________________
Open Web Directory
Reply With Quote
  #3 (permalink)  
Old 02-02-2008, 05:35 AM
WDC's Avatar
WDC WDC is offline
WebProWorld Pro
  
Join Date: Nov 2007
Location: India
Posts: 148
WDC RepRank 0
Default Re: User-agent: *
You need to use allow if you want search engine to index your website.
__________________
Work at home jobs online - Information on work at home jobs.
Sweet love quotes - famous quotes on love and friendship
Reply With Quote
  #4 (permalink)  
Old 02-02-2008, 05:47 AM
WebProWorld Member
  
Join Date: Oct 2004
Location: California
Posts: 42
benc007 RepRank 0
Default Re: User-agent: *
WDC,

Does

User-agent: *
Disallow:

mean that all search engines are not currently allowed to index the website?
Reply With Quote
  #5 (permalink)  
Old 02-02-2008, 06:57 AM
WebProWorld Pro
  
Join Date: Dec 2007
Location: Brussels, Belgium
Posts: 164
Jean-Luc RepRank 2
Default Re: User-agent: *
Hi benc007,

The answer from bobitza is correct.

To allow all robots:
Code:
User-agent: *
Disallow:
To disallow all robots:
Code:
User-agent: *
Disallow: /
The / makes a big difference. The use of Allow: is understood by Google, but it is not recommended as it does not belong to the robots.txt standard.

Jean-Luc
__________________
Checking redirects made easy | | Professional AWStats Services
Reply With Quote
Reply

  WebProWorld > Search Engines > Search Engine Optimization Forum

« Kazazz Search Engine New Unlimited Top Ranking Program | Duplicate sites-problem w/ 301 redirect to site parked on top »
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Changing the referring string in the Browser User Agent whipnet Marketing Strategies Discussion Forum 0 03-07-2006 07:12 PM
You Had Me At Hello; Using Your Blog As Your Job Agent jmiller Internet Industry 11 06-12-2005 12:02 PM
Sober Agent WPW_Feedbot IT Discussion Forum 0 05-07-2005 07:43 PM
User Agent/Crawler: Microsoft and Nokia? Riklaunim Search Engine Optimization Forum 0 09-08-2004 11:32 AM
New Google user-agent? minstrel Google Discussion Forum 0 03-09-2004 01:37 PM


All times are GMT -4. The time now is 08:00 AM.

WebProWorld - Archive - Top

WebProWorld | Advertise | Contact Us | About | Forum Rules | MVP's | Archive | Newsletter Archive | Top | WebProNews
WebProWorld is an iEntry, Inc. ® site - © 2009 All Rights Reserved Privacy Policy and Legal
iEntry, Inc. 2549 Richmond Rd. Lexington KY, 40509

Search Engine Optimization by vBSEO 3.3.0