View Single Post
  #2 (permalink)  
Old 11-28-2008, 11:28 PM
muckle.martin muckle.martin is offline
WebProWorld New Member
 
Join Date: Nov 2008
Posts: 11
muckle.martin RepRank 0
Default Re: Web Crawler frequency and access to the Deep Web

That's an excellent idea and blogs had introduced 'pinging' concept keeping the same problem in mind. I had though about another way to get browser to post the md5 checksum for the page, but that obviously had too many queries going to the search engines.

Sitemaps were probably a step towards finding out if the pages have been updated but obviously they are not as accurate.
Reply With Quote