Google vs AVG toolbar 404 settings

[seopo]

OK here's the dilema.

We provide our sites with bespoke error 404 pages. Google (and maybe others) see's it as an issue if you dont add the 404 code to the header, so we add this info using server side code.

So far so good.

However, recently many people have been installing the latest version of AVG, along with their toolbar. This toolbar detects 404 pages and then serves up its own dodgy search page along with a load of yahoo ads! This looses the visitor from the site and is quite a big issue (especially with moving old sites to new)

So here's was my train of thought.

1) We could turn off the header settings passing the 404 code but Google Webmaster Tools will flag this as an issue. Which one would assume would have an impact on the SERPs
2) We could keep the status quo and forget about the lost traffic
3) Or do some server side scripting to detect bots and serve 404 errors to search engines and non 404 to browsers

Obviously I'm siding towards point 3 but wanted to see if any others had any experience / feedback on this issue?

Cheers,

Richard

[Mel]

I would not worry about the rankings part of things as I really don't think a lack of a proper 404 page will affect your rankings.

But traffic is a greater concern, and in this instance I would simply block AVG from accessing your site in any way at any time.

[Jean-Luc]

One should avoid 404's as much as possible. There are several possible reasons why a page does not exist anymore, but, in many cases, there exists a new page that can be seen as a replacement for the old removed page. In such a case, use a 301 redirect from the old address to the new one. Benefits are obvious: you do not lose traffic, visitors get a page that is close in content to the page they were asking for and search engines will pass the link juice of the old page to the new one.

Jean-Luc

[deepsand]

Originally Posted by Mel

... in this instance I would simply block AVG from accessing your site in any way at any time.

Not possible, as AVG is not accessing the site.

The pseudo-redirection in question is handled via a Browser Helper Object (BHO) installed along with the toolbar. The BHO checks the Header Status Code of each page downloaded; if it's a 404, it does not allow that page to be rendered, but calls for a different page to be loaded & rendered.

[seopo]

We recently moved a large 30,000+ pages site and to a new system and URL structure. In this instance 301's would be too time consuming to do this manually or to create a procedure to automate. So we are stuck with 404's.... I'm going to go ahead and test my third option. Thanks for your replies.

[wige]

The third option won't work.

Simply put, there is no way whatsoever to distinguish a request from a browser using the AVG toolbar and a browser that does not. For security reasons (according to AVG) their toolbar does not identify itself to the server when pages are requested.

I think your best option would be to modify your 404 page so that it redirects users using a 301 redirect to an error resolution page. This will probably get you the best results, as it will allow you to retain users and keep them in the funnel.

To do this, first set your 404 document to a .php document. The code will be the following:

header('Location:http://yourdomain/yourerrormessage',301);exit();

Every time a missing page is requested, the user is redirected to the page containing your error message. You can further add dynamic content to this error message page by analyzing the HTTP_REFERER tag to find possible places to offer to the user as alternative links, or you could tie the page into your site's internal search, and more.

[seopo]

The third option would work if I did it for all browsers though, right? I've noted there are other instances of browser toolbars hijacking the page.

[wige]

Oooh, I misread option 3. Unfortunately, I think there is a chance Google might see that as cloaking. I do know Yahoo does some sneaky checks looking for 404 pages, not sure about Google. However, I think it might be a bit of a risk.

[seopo]

I did think that might be a possibility, but it's almost anti-seo from a search engine perspective.

404's would have a negative impact on your SERPs so I'd hoped it wouldnt be seen as an attempt to improve SEO position by cloaking. What are your thoughts on this?

[wige]

My understanding is that 404 pages can be used as a method to feed search engines alternative links - the page still gets crawled and links in that page will still be followed. I tend to err on the side of caution with things like this, because different SEs may take different approaches. I just think a 301 catch-all, replacing the 404 ErrorDoc might be the safest option, as it would resolve the issue with toolbar pages and would also collect incoming pagerank onto a single page that could then be used to flow that pagerank to appropriate pages on the site.

[seopo]

Intersting. I think that may well be the way forward then. Thanks Wige.

[deepsand]

As I see that wige has beat me to the punch here, I'll just add a few additional observations and a suggestion.

Clearly unconditionally changing the Server Header Response Code from 404 to another is easier to implement and maintain than is a conditional change, which would, at the least, require an ongoing effort so as to maintain a reasonably complete and accurate list of those browser agents which should be/should not be presented a 404.

And, as noted by wige, it wholly avoids the risk of giving the appearance of cloaking.

However, you may wish to consider returning a 302, rather than a 301. This will advise the requester to expect to find the desired content at the original URL in the future, rather than telling it to not revisit the original URL again but to instead find the desired content at the specified new URL.

In this manner, you will retain the option of later re-directing traffic destined for selected individual old URLs to specific new URLs, via a 301, without affecting the content presented to the remaining balance of traffic destined for old URLs.

[honsikrey]

n our research, we have discovered malicious sites returning Http 404 codes (indicating that the web site requested could not be found), and then turning around to serve a page that is filled with maliciousness for display as the error page, sneaky as that may sound. That being said, we do like the idea of having Google’s toolbar prevent the casual surfer from accidentally visiting a malware-laden 404 page.

[wige]

The problem there is that the Google toolbar only displays its own 404 page when the page is smaller than 30k. Most malware laden pages will be larger, causing the bad page to be displayed.