Rob Aronson, Vice President of Search Engine Marketing for Avenue A Razorfish spoke to Murdok for an interview about their Super Intelligent Link Crawler. (SiLC)
Avenue A | Razorfish Launches Link Crawler
Interactive marketing and technology company Avenue A | Razorfish has launched Super-intelligent Link Crawler (SiLC), a tool that crawls Web sites to find errors such as broken links and "404 Error Messages" that inform users the Web page is not loading.
Yahoo Unleashes New Search Crawler
Yahoo Slurp may be showing up heavily in access logs around the web, as the company gave the new spider a workout alongside the old spider before retiring the previous Slurp.
Crawler Alert: Instant Notification When Google Visits
If you want to know the instant Google’s bot visits your site then you’re a severe algoholic. And Crawler alert will help you get your fix. Crawler alert will send you an email when any of the major search bots visit the page you specify.
Crawler Insights From Google and Yahoo!
My favorite sessions at the SES conference were those where Google and Yahoo appeared on the same panels. You could almost always count on some crackling tension between the two search giants.
The “Meet The Crawlers” session was no exception.
Creating a File Content Crawler with ColdFusion
This tutorial will show you how to create a local file crawler that will enable you to find a specified document type (i.e. PDF files) within a directory (and it’s children directories).
Making Sure Your Site is Crawler-friendly
I couldn’t find any “meaty” questions for this issue, so I thought I’d just talk generally about what makes a site “crawler-friendly.” I used to call this “search-engine-friendly” but my friend Mike Grehan convinced me that the more accurate phrase was “crawler-friendly” because it’s the search engine crawlers (or spiders) that your site needs to buddy-up to, as opposed to the search engine itself.