First date with the Googlebot: Headers and compression

Webnauts · #1 (**permalink**) 03-12-2008, 02:14 AM

Google's Webmaster Guidlelines, section Technical Guidelines:

Quote:

Make sure your web server supports the If-Modified-Since HTTP header. This feature allows your web server to tell Google whether your content has changed since we last crawled your site. Supporting this feature saves you bandwidth and overhead.

Source: google webmaster guidelines - Google Search

Do you follow this guideline already? If not, its time to look into that too:
Official Google Webmaster Central Blog: First date with the Googlebot: Headers and compression

Here are a couple tools to check your site if the "If-Modified-Since HTTP header" or "Compession" are implemented and if they work:

For If-Modified-Since:
HTTP Header Checker
or
HTTP Compression and HTTP Conditional Get test tool (beta)

For compression:
Compression Check :: Port80 Software

activeco · #2 (**permalink**) 03-12-2008, 08:21 AM

Yes, such requests should be well supported server side for the benefits of speed and bandwidth savings, thus not only for Google.
I always wondered if Google issues some penalties if server always returns fresh Date, ETag, Expires or Last-Modified headers, while actually providing non-changed document.
Some people would like having Googlebot constantly on their site.

Not only those request and return headers are important.
There is a nice tutorial originally written for Apache's mod_perl, but it explained the major headers in an excellent way.

Webnauts · #3 (**permalink**) 03-12-2008, 08:43 AM

I already use the "If-Modified-Since", "If-Non-Match", "E-Tag", "Deflate", "Gzip", "ContentDigest", etc.

Can you check my HTTP Response and Request Header? Did I probably miss something important?

Thanks.

P.S. My original post issues are required by Yahoo and MSN too.

activeco · #4 (**permalink**) 03-12-2008, 09:14 AM

Apache's default setup correctly handles all usual requests.
So basically, one should take care only not to put any conflicting stuff in <head> section for static pages.
However, dynamic content headers could often be problematic for agents as the headers are generated on the fly and a programmer should properly and carefully deal with it according to language specific practices.

kgun · #5 (**permalink**) 03-12-2008, 09:14 AM

John great post as usual. You give away too much for free.

activeco · #6 (**permalink**) 03-12-2008, 09:22 AM

BTW, regarding dynamic content, a few years ago an user Alex Kemp from another forum was so kind to make a related PHP class.

Webnauts · #7 (**permalink**) 03-12-2008, 09:49 AM

Quote:

Originally Posted by kgun

John great post as usual. You give away too much for free.

Well since people do not want to pay, I do not have another choice.

Webnauts · #8 (**permalink**) 03-12-2008, 09:50 AM

Quote:

Originally Posted by activeco

BTW, regarding dynamic content, a few years ago an user Alex Kemp from another forum was so kind to make a related PHP class.

Thanks for sharing. I passed it to my chief developer for review.

Webnauts · #9 (**permalink**) 03-12-2008, 12:13 PM

Does anyone here exclude the possibility that through the implementation of the "If-Modified-Since" HTTP Header, the bots might visit and crawl a site more frequently?
I am testing this already, but if someone knows already, please drop a note here.

Thanks.

bobitza · #10 (**permalink**) 03-12-2008, 02:01 PM

Interesting topic. So far I only knew that bots can crawl gzipped xml sitemaps. Anyways, what's the benefit of all this? For compression ... save B/W? For if modified since to manipulate crawling rate?

I tested a page with one of john's tools ... I got: Last-Modified: Mon, 03 Mar 2008 17:29:42 GMT . We're still on March, is this working?

**mjtaylor** · #11 (**permalink**) 03-12-2008, 02:16 PM

Quote:

Originally Posted by kgun

John great post as usual. You give away too much for free.

Great post and it is not possible to give away too much for free. Sharing is kind of like tithing, the more you give the more comes back to you!!

Thank you, John!

**incrediblehelp** · #12 (**permalink**) 03-13-2008, 01:18 AM

Quote:

Originally Posted by Webnauts

Well since people do not want to pay, I do not have another choice.

lol, good stuff john

Webnauts · #13 (**permalink**) 03-13-2008, 01:37 PM

Quote:

Originally Posted by bobitza

Interesting topic. So far I only knew that bots can crawl gzipped xml sitemaps. Anyways, what's the benefit of all this? For compression ... save B/W?

Gzip or Deflate compress your files and speeds up the loading time of your pages.

Quote:

Originally Posted by bobitza

For if modified since to manipulate crawling rate?

Logically should be so. I strongly believe that is can manipulate the crawling rate.
I am now running a live test with my site and with some customers sites that did not have that implemented yet, and I should have some facts soon.

Quote:

Originally Posted by bobitza

I tested a page with one of john's tools ... I got: Last-Modified: Mon, 03 Mar 2008 17:29:42 GMT . We're still on March, is this working?

The tool is working so far I am aware of. To make sure, run a test with this tool HTTP Header Checker and then with ours SEO Workers Search Engine Optimization Analysis Tool and see if there is any difference. If there is no difference, could it be that you don't understand the the meaning of the Last-Modified?

Webnauts · #14 (**permalink**) 03-13-2008, 02:32 PM

Quote:

Originally Posted by incrediblehelp

lol, good stuff john

Well it is true man.

I began to believe that people cannot make any difference between the abbreviations SEO and SSA (Social Security Administration).

activeco · #15 (**permalink**) 03-13-2008, 02:52 PM

Quote:

Originally Posted by Webnauts

I began to believe that people cannot make any difference between the abbreviations SEO and SSA (Social Security Administration).

How about SOE?

Webnauts · #16 (**permalink**) 03-13-2008, 03:46 PM

Quote:

Originally Posted by bobitza

For if modified since to manipulate crawling rate?

I mentioned before that I am observing this issue. I have setup this page if someone is interested to follow:
Google Crawl Cycle

OFF TOPIC:
I am also using that page to test a new SEO hack I came up with.

I am not using any robots meta tags with noindex and nofollow, and I do not disallow the bots to access it with robots.txt. And no kind of cloacking!

That is even more interesting. Don't you think? If that hack works, the page cannot be indexed, and not followed!

If I have used the robots.txt, still it would have been followed, and it could pass PR juice if the page had links pointing to other pages or sites.
More about: SEO Linking Gotchas Even The Pros Make | Andy Beard - Niche Marketing

God bless SEO.