Sitemaps won't Verify w/ 404 redirect

[blitzen]

Hello,

I'm using a cgi program called from htaccess that when a 404 is detected, the program writes a page to the browser that redirects the user to the home page.

Google Sitemaps won't verify the site because it thinks it knows what it's doing. It says the page is returning an okay status even though it's a 404.

I'm using the empty file in my ht root directory to verify sitemaps instead of adding the html code.

I use several mod-rewrites and place the call to the external program at the end of the rewrites.

How can I fool G Sitemaps so that my site will verify and still redirect pages when a 404 is detected?

[ackilles]

You disable your 404 cgi program during the process of verification then enable it again or just use the meta tag that Google supplies.

[incrediblehelp]

Can you post a 404 error page? Are you sure your 404 page is showing a 404 error code in the header and not a 200?

[EditFast]

I use a script called Guardian for the same purpose which allows me to modify the 404 page to look like your home page or you can redirect to the home page itself. There are other useful functions as well such as sending you a notification of errors on the site when a user tries to access a non-existent page, or does not have permission to access the page etc. I have no affiliation with this script but I have used it for years and recommend it highly. There is a freeware version but the registered version is just $20.
http://www.xav.com/scripts/guardian/

[blitzen]

IncredibleHelp:
The page really doesn't exist, 404. Can I fool it to return a 200 in the header instead of 404?
Would this work?
ErrorDocument 404 http://www.mysite.com/

Ackilles:
You're correct. If I disable the cgi script, Sitemaps will verify the site. I did this in the past, but when I logged in recently, the site wasn't verified. I don't know how that happened because nothing changed in my Sitemaps settings. I assume that they changed something in their alg.

EditFast:
That's a good idea. Your method sounds like what I'm using but it instead of a home-page look-alike, mine writes an HTML page that redirects to my home page. With the redirect, I don't have to remember to change that script everytime the home page changes.

That brings up another question.
Would Google look at it as a duplicate page, either with your method (home-page look-alike) or mine (redirect to home page) or "ErrorDocument 404 http://www.mysite.com/"?

[brandrocker]

To verify your site in this situation, it is better to use META TAGS. I had similar issue with Joomla site.
I used META TAG to verify my websites.

[olwen]

If you are redirecting to your sitemap I think you may find a 301 (permanently moved) response works best for you. Assuming your redirect CGI routine is in PERL it would look something like:

#!/usr/bin/perl -w
use strict;
print "Status: 301 Moved Permanently\n";
print "Location: http://somewhere/page.htm\n\n";
exit;

I haven't used PERL very much, and not recently. I had to Google it and found this code snippet at http://www.somacon.com/p145.php

[brucet]

We also use meta tags to verify our site, because Google can't seem to cope with an error page set using 'ErrorDocument 404' in .htaccess.

It sucks IMO, because any site worth its salt should have a proper error page, and this is the standard method for implementing one on a Linux server. But I guess there are technical reasons for it, and the meta tag verification does work perfectly well as an alternative.

[wige]

When you set the error handler as ErrorDocument 404 /somefile.html, the server will send the browser a 404 message and the contents of the somefile.html page. When the handler is set as ErrorDocument 404 www.somesite.com/somefile.html, the server sends the browser a 302 redirect to the specified address, even if the file is on your server. As a result, Google and other spiders do not know why they are being redirected.
If you redirect to your home page, it is not duplicate content because Google sees the actual home page address. If you make the error page look like your home page, then it can be duplicate content because that content shows up with the incorrect page URL.
Also note that this can be confusing for visitors - imagine you are on a web site and click a link (that you don't know is broken, but it is) and you are just dumped back on the home page. If the customer is in the process of ordering or has browsed deeply into your site, they may get discouraged and leave.

[okparrothead]

I can create error pages using CPanel with the .shtml extension. Blitzen, is this the script you're using? Seems I can add any code I want to the error page. Including a redirect I suppose.