Web Crawler frequency and access to the Deep Web

[CraigH]

Hi all,
I am working on a personal project looking at web crawler frequency and the accessibility of the Deep Web to crawlers. From a webmaster's point of view, the problem as I see it is getting content into a search engine's index quickly after publication, and getting exposure to deep-web content within the search engines.

The project website is at Site Update Notification where I have tried to describe the problems and a solution based on an idea of web server agents notifying crawlers that content has changed (preventing unnecessary re-crawling and use of your bandwidth, too!).

I'd be really interested to hear everyone's views on the problem domain and the described Site Update Notification system. It is technically feasible (I have some prototypes working nicely which I may open-source if the project has significant interest), but would the webmaster community embrace the technology if major search engines were involved? Any feedback, suggestions or general comment would be great.

Thanks
-- Craig

[muckle.martin]

That's an excellent idea and blogs had introduced 'pinging' concept keeping the same problem in mind. I had though about another way to get browser to post the md5 checksum for the page, but that obviously had too many queries going to the search engines.

Sitemaps were probably a step towards finding out if the pages have been updated but obviously they are not as accurate.