How to Create Sitemap for Large Websites & Dynamic Websites?

[tushar123]

Hi All,

Recently I encountered 2-websites, one is generating the pages dynamically as it's an e-commerce website, and the other one is a very large website consisting of some nearly 4,000 odd pages.

Now there is a big question for me, how to create a Sitemap for these websites. And I am a bit puzzled as how to create the HTML sitemap also, along with the Google XML sitemap.

Considering the HTML version of the sitemap, is it recommended that I should put some odd 500 pages in the sitemap?

Please do suggest some ways for,
- creating google sitemap for the dynamic and the large website, &
- creating the html sitemap for the large website.

Thanks in advance.

[Bagi Zoltán]

Some times ago there was a recommendation in the Google Quality Guide to use 100 as a limit for the number of links on a given webpage, have no idea if they still suggest this or it has been changed since that time.

Considering that you have 4000 pages I think you should make display only the product category structure in the static (html) sitemap which would ease the navigation for the visitors.

Regarding the xml sitemap I think you may want to read this article about creating xml sitemap without using fopen.

[chandrika]

For sitemaps such as url lists and xml, you can put 50000 urls per page of the sitemap, more than that should be split and an index made.

But I think that an html sitemap, as Bagi Zoltan said, being html pages, Google does reccomend 100 links per page, which generally refers to normal site pages, but as an html sitemap is more for visitors use, it may still apply.

I use the sitemap generator provided at
Create your Google Sitemap Online - XML Sitemaps Generator for a large site. I find it good, it creates xml, html, ror and url lists and you can specify how many urls per page for the html sitemaps and xml as well. So you could split a large sites urls over several pages in the html sitemaps using that.

[JasonJersey]

For sitemaps such as url lists and xml, you can put 50000 urls per page of the sitemap, more than that should be split and an index made.

But I think that an html sitemap, as Bagi Zoltan said, being html pages, Google does reccomend 100 links per page, which generally refers to normal site pages, but as an html sitemap is more for visitors use, it may still apply.

I use the sitemap generator provided at
Create your Google Sitemap Online - XML Sitemaps Generator for a large site. I find it good, it creates xml, html, ror and url lists and you can specify how many urls per page for the html sitemaps and xml as well. So you could split a large sites urls over several pages in the html sitemaps using that.

Your right! XML Sitemaps is a great tool. If your site is a blog, then you may consider a Wordpress Sitemap Plugin.

[mark.smith]

I have worked on various e-commerce websites. Generally used this site: http://www.xml-sitemaps.com ( also suggested by Chandrika ) and really found it works for me.

[cbosleeds]

Looks like we have a strong consensus here. I go by the limit of 100 pages for an html sitemap aswell, and I also use Create your Google Sitemap Online - XML Sitemaps Generator as my preferred sitemap generator. I'd also agree that having a sitemap that goes by categories would be the right way to split this down, you could even go further and have sub-sitemaps for each category, maybe.

[newsblaze]

When you generate the sitemap, check that all the URLs are unique.
In ecommerce sites, there is often more than one way to get to a product, with a different URL, so be sure that for any particular page of content, there is only one URL.

You won't have any problem with 4,000 pages in the sitemap, as Chandrika said, you can have 50,000.

LOL, if you think 4,000 is very large, what would you call mine with 640,000 ?

I generate my sitemap with custom software I wrote, that walks directory trees and manufactures the correct URLs. Some sitemap generators make HTTP requests that will be seen in your webserver logs.

Regarding HTML sitemap, 100 isn't a hard limit, but if you are using that for visitors to read, breaking up the list is a nice thing to do. The suggestion of grouping categories by Bagi and cbos is a good one.

[AjiNIMC]

Creating sitemaps is really easy. You need to create a sitemap index, that can hold many sitemaps inside it. Example: http://www.upad.co.uk/sitemap.xml, now you can create an html sitemap too like UPAD | SiteMap | Reinventing Renting

This site will have over 5,00,000 pages, so we will be having around 10 xml sitemaps, each containing 50k links in it. The html sitemap pages will not have more than 100 links per page.

It is not very difficult to program as a large website will always have cms (any apps to manage the web pages using a DB).

Do let me know if you need more help.

Regards,
Aji Issac

[subhzash]

There are two solutions:-

1. Create your own sitemap with the help of python which you can install in server by which you can generate sitemap for unlimited pages

2. Otherwise you need to go the third party tools like Create your Google Sitemap Online - XML Sitemaps Generator but you need to take the paid version for generating more than 500 pages. You can get the static sitemap also


Regards
Subhzash

[tushar123]

When you generate the sitemap, check that all the URLs are unique.
In ecommerce sites, there is often more than one way to get to a product, with a different URL, so be sure that for any particular page of content, there is only one URL.

You won't have any problem with 4,000 pages in the sitemap, as Chandrika said, you can have 50,000.

LOL, if you think 4,000 is very large, what would you call mine with 640,000 ?

I generate my sitemap with custom software I wrote, that walks directory trees and manufactures the correct URLs. Some sitemap generators make HTTP requests that will be seen in your webserver logs.

Regarding HTML sitemap, 100 isn't a hard limit, but if you are using that for visitors to read, breaking up the list is a nice thing to do. The suggestion of grouping categories by Bagi and cbos is a good one.

LOL 640,000 is really BIG..