Re: Web Crawler frequency and access to the Deep Web
That's an excellent idea and blogs had introduced 'pinging' concept keeping the same problem in mind. I had though about another way to get browser to post the md5 checksum for the page, but that obviously had too many queries going to the search engines.
Sitemaps were probably a step towards finding out if the pages have been updated but obviously they are not as accurate.
|