I have a reasonably sized site.
http://www.governmentsecurity.org, I have been performing a lot of optimizations on my content and wanted to create and submit a google site map. So I installed python on my server and configured the google site map generator. My config lists the root directory of my site, 2 web pages, one for the main site, the other for the forum and I directed the site map generator to scan my access_log.
Well it started running and I left it running, I check back 4 hours later and noticed it was still running, the site map generator stated that it had created 6 site map files because a limit of 50,000 urls had been reached on each one. Now my site is big but not that big. But I figured I would let it run so I did. In the morning I noticed that my site was un responsive, so I ILOed into it and restarted it. Checking the logs, it turn out the Site map generator had locked it up. Has anyone noticed this behavior, or know a way to generate a sitmap for a site with 8,500 html pages and about 119,000 forum posts?
I also have a google
PR of 6 and I hope this does not cause some problems with that if there was a partial submition by the tool.