Talked to MS techs
I talked to a couple of MSN techs at SES in Chicago this week. I asked them about their bot using so much bandwidth and they said it is partially intentional. They want to have the cached version as fresh as possible for the best user experience. They feel that if someone clicks on a page and it doesn't show what their excerpt says it will show, that is a poor user experience. Instead, they're focusing on crawling the pages listed for frequent searches as often as multiple times daily. He said there would be details for telling the MSN Bot how often your content updates posted on their website when their search goes live, but until then they're going to try to figure out how often your content changes through their crawler and start to make adjustments at smarter intervals.
Brian.
|