Submit Your Article Forum Rules Search
WebProWorld
Register FAQ Calendar Mark Forums Read
Search Engine Optimization Forum SEO is much easier with help from peers and experts! The WebProWorld SEO forum is for the discussion and exploration of various search engine optimization topics. Any non (engine) specific SEO or SEM topics should go here.

Share Thread: & Tags

Share Thread:

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 07-23-2007, 07:24 PM
WebProWorld New Member
 
Join Date: Jul 2007
Location: Dallas. TX
Posts: 8
Monkey SEO RepRank 0
Default robots.txt and indexed pages

i created a robots.txt file sometime last week for my blog to restrict the duplicated pages created by wordpress. i think i did it right. i tested it in google webmaster tools and it shows me that the pages i want restricted are in fact restricted by my robots file. here's what is in it:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
Disallow: /cgi-bin/
Disallow: /2006/
Disallow: /2007/
Disallow: /*?*
Disallow: /page/
Disallow: /iframes/
Disallow: /recommends/
Disallow: /category/
Disallow: /*/feed/

webmaster tools tells me it last downloaded my robots.txt file at 9 am this morning, i know that it has downloaded it at least a couple more times prior as well. but some of the pages are still in google's index:
  • www.monkeyseo.com/category/google/
  • www.monkeyseo.com/2007/03/
for example. there are more in the index as well.

how long will it take google to de-index the urls restricted by my robots.txt file?
Reply With Quote
  #2 (permalink)  
Old 07-24-2007, 11:28 AM
stymiee's Avatar
WebProWorld Veteran
 
Join Date: May 2006
Location: New Jersey
Posts: 429
stymiee RepRank 0
Default Re: robots.txt and indexed pages

It depends on their crawl rate of those pages. If you want them removed sooner you can remove them through your webmaster control panel.
__________________
John Conde
Brainyminds Merchant Account Services eBook Giant
Reply With Quote
  #3 (permalink)  
Old 07-26-2007, 08:02 AM
amar's Avatar
WebProWorld Pro
 
Join Date: Aug 2005
Location: India
Posts: 295
amar RepRank 0
Default Re: robots.txt and indexed pages

Submit unwanted pages to Google link removal tool in Google webmasters tools.
__________________
SEO Professional India
Reply With Quote
Reply

  WebProWorld > Search Engines > Search Engine Optimization Forum

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
PR3 but no indexed pages buddhu Google Discussion Forum 15 02-01-2007 06:25 AM
Not all pages being indexed jkjazz Search Engine Optimization Forum 3 06-09-2005 05:10 PM
How many pages are really indexed? jimvet Search Engine Optimization Forum 1 06-06-2005 05:54 PM
Indexed Pages C French Yahoo! Discussion Forum 3 05-12-2005 12:19 AM
Blind Robots or Pages hiding from view seniorsmile Search Engine Optimization Forum 5 02-18-2004 11:06 AM


All times are GMT -4. The time now is 03:51 PM.



Search Engine Optimization by vBSEO 3.3.0