WWW vs non-WWW - understanding the physical file and directory background

[deepsand]

DS - Well, the dichotomy between the crawler showing "Robots Found" and the "Fetch as Googlebot" showing "Missing robots.txt" obviously indicates a flaw in GWT which didn't previously exist. The messages saying that pages are "unreachable" and pointing to possible server response and delivery problems are worrying, if they are to be believed. If the client is indeed on a slow or dodgy server, I can obviously recommend that they move, but in working with the site, all of the pages load instantly, and I know that they are small, html and lite-image pages. When you speak of "lack of database synchronization", "distributed database with dynamic load balancing", I assume you're referring to Googles Database / load-balancing etc.

Correct.

With two canonical forms, and two upload channels for each - "Fetch as Google" and sitemap upload - it would not be unlikely that four different Data Centers are involved. And, that's not even considering the possibly of load swapping if one or more get bogged down.

Theoretically, the Caffeine platform was to minimize any DC being overloaded. However, the demands of Panda for the reevaluation of huge amounts of data cannot not have put an unexpected strain on the DCs. IMO, until Panda is well behind us, all Google data will be in an elevated state of flux.