Sitemap duplicate URLS

[emersonworldwide]

I have used the Google SiteMap generator and had it validated. I cannot find an answer anywhere to these questions:
Before I submit it, do I need to go into the xml code and delete:
1. duplicate URLs
2. a page that we no longer link to on our site
3. pages that do not need to be spidered like Guarantee.htm; ReturnPolicy.htm, etc, or do I just list these exclusions in the robot.txt (which I have yet to create).
4. What is the significance of some of the URLs being in bold and most not being in bold?

I thank any and all who can answer these questions for me.

[TrafficProducer]

The Best

is

GSiteCrawler will help you generate the best Google Sitemap file for your website. The GSiteCrawler uses different ways to find all the pages in your website and can generate all sorts of files, statistics and more. The sitemaps file format has lately been also adapted by Yahoo! - even MSN/Live.com is pledging it's support.

Google Sitemap Generator for Windows :: GSiteCrawler

This will Crawl your pages... You may tell it to ignore pages as you wish, etc..

Able to create more sitemaps HTML, Google, Yahoo, ROR etc...

Other Info:
Google Site Map
Yahoo Site Map, Yahoo Site Explorer
Site Maps

[Webnauts]

Originally Posted by TrafficProducer

The Best

is

GSiteCrawler will help you generate the best Google Sitemap file for your website. The GSiteCrawler uses different ways to find all the pages in your website and can generate all sorts of files, statistics and more. The sitemaps file format has lately been also adapted by Yahoo! - even MSN/Live.com is pledging it's support.
VIGOS Gsitemap - Free Google Sitemap Generator for Windows

This will Crawl your pages... You may tell it to ignore pages as you wish, etc..

Able to create more sitemaps HTML, Google, Yahoo, ROR etc...

Other Info:
Google Site Map
Yahoo Site Map, Yahoo Site Explorer
Site Maps

Does GSiteCrawler obey to the robots.txt? I did not have a so good experience so far.
I am now using VIGOS Web Acceleration and HTTP Compression Software
It is easy to use and I love it.

[TrafficProducer]

Does GSiteCrawler obey to the robots.txt?

To be honest I'm not 100% sure about this... I know it is able to check locally and remote links and able to produce robots.txt from the banned list you tell it..

So far as I know it's free to use.

Personally it can be a bit of a nuisance to set up don't use links, (at first), etc but once done the project/s may be saved for later use..

Includes FTP to your host and Ping Google/Yahoo. and also make compressed . gz files, and others...

(I use it for HTML site maps as well).

FAQ:=FAQs and Documentation :: GSiteCrawler


All I can say it's the best I've found so far

I am now using VIGOS Web Acceleration and HTTP Compression Software

This looks to be more about compression of HTML??? and not so much SiteMaps???

[Webnauts]

Sorry buddy. I just corrected the link above: VIGOS Gsitemap - Free Google Sitemap Generator for Windows

[TrafficProducer]

The name seems almost the same as GSiteCrawler.

[hyperdog]

I recommend you keep as many pages indexed as possible. Be sure to redirect old pages if you just can't think of anything else to do with them. Your guarantee page can certainly be optimized for potential customers seeking a guarantee with one of the products you sell. Granted, it isn't a high traffic situation!

[Webnauts]

Originally Posted by hyperdog

I recommend you keep as many pages indexed as possible.

Can you explain why?

[craigmn3]

Site Map Generators come in many flavors, some I have used got caught in bread crumb navigation loop and kept repeating the same links. A better sitemap generator would do better for you

As to the pages you don't link to......somewhere on your site...you do link to it. These things can't pull links out of mid air....they follow them.

Always designate the pages you don't want crawled in your robots.txt

and the bold vs un bold is some aspect of the program you are using, they mean nothing in google sitemaps.

The besst freebie site map generator I have run into is:

Create your Google Sitemap Online - XML Sitemaps

I think they have a 500 page limit so for the bigger sites it's not very useful

[kgun]

Originally Posted by emersonworldwide

I have used the Google SiteMap generator and had it validated. I cannot find an answer anywhere to these questions:
Before I submit it, do I need to go into the xml code and delete:
1. duplicate URLs
2. a page that we no longer link to on our site
3. pages that do not need to be spidered like Guarantee.htm; ReturnPolicy.htm, etc, or do I just list these exclusions in the robot.txt (which I have yet to create).
4. What is the significance of some of the URLs being in bold and most not being in bold?

I thank any and all who can answer these questions for me.

If Google is consistent, overlapping information and broken links should be filtered out. But I am unsure.

May be the best is to make your own XML sitemap. It is explained in detail in the following book:

Thomas Meyer (last edition): "No Nonsense XML Web Development With PHP"

You get the necessary code with the book.

There are various methods to transform an XML file to a new XML file, where you have automatically eliminated the information that you don't want in the target XML sitemap.

• The simplest is an XSL(T) stylesheet, that may be all you need.
• The most advanced, where you have (nearly) full control of the output is using PHP DOM functions.

[Orion]

yes GSiteCrawler (if you check the box) will obey your robot.txt file then create your xml site map based on those settings.. it will also create a yahoo sitemap for you!

great product I've been using it for 2 years or more now.

[DoneInStyle]

Most of the better sitemap generators allow you to set filters. Most of the dynamic programs use similar conventions in the urls that allow you to figure out filters you can use. The better ones also respect robots.txt so you can stop hits on your admin access pages and other restricted areas.

[emersonworldwide]

Thank you for this information. I will have to "sift" through it for my best options. But, now I have another question on same topic:
I have 10 strange URLS in the sitemap that all link back to my index.html page and I do not know where they originate! They all show the ~ mark????:
/index.html?Elbow.htm~Documents
/index.html?Gloves.htm~Documents
/index.html?knee.htm~Documents
/index.html?Lumbar.htm~Documents
/index.html?wrist8.htm~Documents
/index.html?wristProds.htm~Documents
/index.html?wristsupport2.htm~Documents

[kgun]

You know this page:

https://www.google.com/webmasters/to.../protocol.html

More precisely this:

Is there an XML schema that I can validate my XML Sitemap against?

https://www.google.com/webmasters/to...faq_xml_schema

with a link to the following XML Schema that defines the xsd name space.


That is the professional way to validate a well-formed xml sitemap document.

P.S. You do not need to go into the name space thread at W3 Schools. It is there if you need deeper information.

[DoneInStyle]

That url form looks like the kind of bird tracks in the url made by a dynamic program. If no other urls in your sitemap have any part of that end structure, then you may be able to use a portion of it for a filter. I'd suggest trying ~Documents first as a filter and see what happens.

[heavener]

Quoted:
"I recommend you keep as many pages indexed as possible."

"Can you explain why?"

The last thing a website needs, IMHO, is to shut the door on even a single customer/visitor. They might be the one in a hundred guy whose performance hangs in the balance and is willing to reward the organization that helps.

Example: I worked with a VP of Marketing to develop a commercial website. He wanted it to "look just like Moen Faucet". Why? Because Moen replaced a busted faucet for free when he got their phone number off the website. Note that he didn't use the website to make the support request, just got the phone number. And yet he bought all Moen faucets for the house he built later because "they have the best website in the world."

Don't discredit parts of your site that may admittedly not bear high traffic - they might be exactly the mine of gold one of your customers needs. I can't tell you the number of sites I've been to where information is so buried it's useless. Let it be indexed and don't worry about it.

P.S. - I once found a company's contact address buried in the middle of their privacy policy page - the only place on their whole site (I had to use QuadWeb Sucker to grab the whole site before I could find it).

Michael Heavener
heavener@heavenr.com

[NYChris]

Does anybody know if this thing can generate from daily logfiles?
Currently, I use the python script from Google to generate my sitemaps from my log files.
I have to do it this way because the site is mostly database driven and contains nearly 100,000 URL's.

I checked out this GSiteCrawler you are talking about and I don't see an option to generate from a log file.
However, I do see that it has a 1GB limitation for the DB and that raises another flag because my daily log files are very close to that size.

Chris
NYC Real Estate
NYC Real Estate Brokerage with Listings of Manhattan Apartments & Lofts for Rent & Sale

[Tolis]

We have the site easytraveller.gr and of the first to use GSiteCrawler since then they have been many releases each better from the previews. I think this is the best way to make a sitemap. Before you create the sitemap you can put your terms of type of files you do not want to be included and as the crawling is finished you can edit from the URLs tab to remove duplicate urls and also fix the frequency of changes

[emersonworldwide]

Thank you all for this information! I am ready to go from here!

[DoneInStyle]

Let us all know what is working for you. Others looking for the information at some future date may possibly benefit.

[zycon5000]

Originally Posted by NYChris

...I checked out this GSiteCrawler you are talking about and I don't see an option to generate from a log file.
However, I do see that it has a 1GB limitation for the DB and that raises another flag because my daily log files are very close to that size.

GSiteCrawler is an excellent program. We use it frequently to generate our sitemaps. If you have a concern about size, download their SQLDB version if you have that installed on your machine. Unfortunately, they don't have the new beta version in SQL format, only Access format. The 1GB limitation is probably an Access limitation, not the application limitation. Also, GSiteCrawler even found some duplicate URL's while crawling and automatically excluded those from the sitemap.