|
|
||||||
|
||||||
| Index Link To US Private Messages Archive FAQ RSS | ||||||
| Webmaster Resources Discussion Forum Sitemaps and robots and logfiles -- Oh My! If you have any questions, comments, concerns and/or ideas about the tools currently available to webmasters to make their lives... 'easier'. Here's where you need to be. Know of a good tool? Post it here. Got something funny in your logfiles? Maybe we can help. |
Share Thread: & Tags
|
||||
|
![]() |
|
|
LinkBack | Thread Tools | Display Modes |
|
||||
|
You Can't go wrong with GsiteCrawler
Google Sitemap Generator for Windows :: GSiteCrawler It finds everything. |
|
||||
|
Quote:
__________________
"Being an expert isn't telling other people what you know. It's understanding what questions to ask, and flexibly applying your knowledge to the specific situation at hand. Being an expert means providing sensible, highly contextual direction." Jeff Atwood SEO Workers - Search Engine Optimization Consulting Company | SEO Analysis Tool | Webnauts Net SEO |
|
||||
|
I'm sure enough.
Feel free to recommend an alternative! |
|
||||
|
So you are saying that it will obey to:
1. X-Robots of an .htaccess file? 2. Noindex and Allow robots.txt directives? 3. Noindex and Noarchive meta robots directives? 4. Wildcards in Robots.txt? Correct? About an alternative? I do not know any. That is why I am asking.
__________________
"Being an expert isn't telling other people what you know. It's understanding what questions to ask, and flexibly applying your knowledge to the specific situation at hand. Being an expert means providing sensible, highly contextual direction." Jeff Atwood SEO Workers - Search Engine Optimization Consulting Company | SEO Analysis Tool | Webnauts Net SEO Last edited by Webnauts; 06-15-2009 at 01:15 PM. |
|
||||
|
In cases where it does list files you don't want in the sitemap, simply go through the list of files and uncheck them to exclude them.
I've sent a message asking the creator to explain how to set GSiteCrawler up to follow the current robots.txt, .htaccess files and meta tag directives such as "noindex" and "nofollow". I'll post his response later. |
|
|||
|
Quote:
|
![]() |
|
| Thread Tools | |
| Display Modes | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Pls help me for Sitemap generator | wqamar | Search Engine Optimization Forum | 4 | 03-19-2009 07:33 AM |
| Sitemap generator | duongo75 | Search Engine Optimization Forum | 3 | 10-08-2007 06:32 AM |
| A good free sitemap generator that works for MSN Search. | WebMasterKrames | Search Engine Optimization Forum | 2 | 02-09-2007 12:18 PM |
| where to get a good sitemap generator | lukkyjay | Google Discussion Forum | 18 | 12-18-2006 10:08 AM |
| Freeware Sitemap generator | No Clue | Search Engine Optimization Forum | 3 | 06-20-2005 11:53 AM |
|
WebProWorld |
Advertise |
Contact Us |
About |
Forum Rules |
MVP's |
Archive |
Newsletter Archive |
Top |
WebProNews
WebProWorld is an iEntry, Inc. ® site - © 2009 All Rights Reserved Privacy Policy and Legal iEntry, Inc. 2549 Richmond Rd. Lexington KY, 40509 |