Hi all,
I am working on a personal project looking at web crawler frequency and the accessibility of the Deep Web to crawlers. From a webmaster's point of view, the problem as I see it is getting content into a search engine's index quickly after publication, and getting exposure to deep-web content within the search engines.

The project website is at Site Update Notification where I have tried to describe the problems and a solution based on an idea of web server agents notifying crawlers that content has changed (preventing unnecessary re-crawling and use of your bandwidth, too!).

I'd be really interested to hear everyone's views on the problem domain and the described Site Update Notification system. It is technically feasible (I have some prototypes working nicely which I may open-source if the project has significant interest), but would the webmaster community embrace the technology if major search engines were involved? Any feedback, suggestions or general comment would be great.

Thanks
-- Craig