Search Engine Demerits for 404's?

[Judith]

Hi...I'm brand new here. I've just launched my first commercial site, completely replacing an already existing site.

I am seeing 404 errors in my web stats that indicate that engines are looking for the old pages. Will we be penalized for these error pages? I understand that oversubmission of sites is punishable by banishment to Never Never Land. What is the best way to handle the problem?

Thanks for your help.

[cyanide]

Well I don't know about 404 penalizing per se,
but are you using custom 404 pages?

If you're not, then the robots have nowhere to go.
If you are, atleast the robots will continue to crawl other pages of your site, via links from the custom 404 page.

[TheWebDoctor(tm)]

Before I can answer this question, I need to clarify some things.

404 error pages can be useful!

If you are attempting to remove old pages that are no longer used, ensure those pages are not being directed to your main page. By ensuring that they are not being redirected to the new page you are informing the search engines that the page needs to be removed from their directory.

If you are attempting to rename your pages then you need to ensure you are using the redirect method allowed for your server. This will inform the search engines that the page has been renamed and their index will reflect that.

The only problem you have left is links coming to the old pages. If the pages are no longer used the links will be broken and not allow people to get further into your site.

A custom 404 page will allow people and search engines to navigate your site. This can be advantageous to you, your visitors, and the search engines.

Because the page provides the HTTP 404 indication, the search engines know the page no longer exists. Some search engines will remove the old page.

Those are all the ways to handle the situation.

I have not seen any search engine that penalizes a site for 404 errors. I redesigned a site and did not use a custom 404 page. This immediately notified the search engines that the old pages did not exist any longer. They removed the old page. I've actually done this on many sites without it hurting the site.

I hope this helps.

[Judith]

Why didn't I think of that! It was too obvious.

Thank you both very much. I didn't create the old site and it no longer exists, so so I can't create custom pages for each deleted page (hindsight says I should have printed a site map).

But I did create a custom page that will at least keep visitors in the site instead of sending them to my web host. Thanks so much!

[rlrouse]

The best approach to take is to simply never completely remove a page from your site. Ever. Period.

It's far better to change the page with new relevant content. After all, you already have a page that is probably listed in the search engines, and it could possibly have some inbound links to it as well.

Why throw those benefits away by removing the page when all you have to do is change the content? Never underestimate the power of recycling!

[DrTandem1]

The 404 errors will show you which pages are be requested and not found. It appears that you have already determined that these are your old pages.

In this situation, it is best to place a redirect in the meta tags that will automatically send your visitor to the new page you would rather have them view. You can vary the delay so you can post a message on the old page that tells the visitor what is happening and give them the link to the new page, if you want.

If you make the added redirect immediate without bothering to change the content of the old page, the visitor may see a very quick glimpse of the old page as they are being redirected.

The importance of using the redirect is so that the search engine will not receive a 404 error. If they do, it may hurt your ranking. Some engines will also penalize you for the redirects. However, the redirect is only temporary. While adding the new meta tag you should also insert a meta tag for the robots not to index nor follow the old page. I would remove the redirect after 30 days.

It is possible that visitors are following old links from other sites. You web logs should be checked to see the referrers. If they are other sites, try contacting those sites to update the URLs.

If at all possible, re-use the old page URLs and simply use the new content. I know this may not be ideal, if the content of the page is completely different. If the content is substantially different, (say you were a hotel and the page was called rooms.htm and now you're a rental car company and the page is cars.htm)I would use the redirect page with a delay that gives the visitor time to read a message telling them what is happening.

Lastly, if you don't want to use redirects or change the old page's content to show a new link to the new pages of the site, you should consider a custom 404 error page. I think all sites should do that anyway. It then gives the visitor some options other than their browser's Back button.

[dealercrm]

I would start by analyzing your web logs. Take a listing of all the 404 error pages and create pages with those missing names. Continue this until no more 404 errors exist.

The pages can be anything, but yes content that was their before would probably be best to stick by. However you don't know what was there so just embrace the traffic that is coming to these pages.

It is important to note that your 404's are generating traffic. So embrace it to market what you currently are working on. Redirects are bad. Content is good.

Hope that helps. I know some of this is repeat but just my opinions.

[Judith]

I did post a custom error page. All good advice about not ever getting rid of old pages or at least recycling them. However, it's too late. Someone else created the old site and it is gone forever.

Most of the errors are coming from a gallery of photos that no longer exists. There are photos with each page now. Think I'll follow dealercrm's suggestion and create a mini-gallery perhaps with links on it to each of the sections of the new site.

Thanks for keeping me thinking on how to solve this problem.

[paulhiles]

One problem I've experienced when trying to analyse log files for 404 errors is that some of the entries are from hackers or 'script kiddies' that are just trying to locate a commonly used mailform script. So in actual fact, although it looks as though they're requesting pages that don't exist.. they are not strictly speaking 404 errors.

Paul

[Conficio]

Hi Judith,

I didn't create the old site and it no longer exists, so so I can't create custom pages for each deleted page (hindsight says I should have printed a site map).

But I did create a custom page that will at least keep visitors in the site instead of sending them to my web host.

Just a custom error page is fine, but not enough. As some fellow here mentioned, there are old links, old search engine entries and last but not least old bookmark entries out there. You should always keep those old pages for that reason and redirect accordingly.

If you don't know the old site, how about looking it up at the "Wayback machine" http://www.archive.org/web/web.php
If it existed long enough you whould be able to retrieve it there or check what pages where valid and what was their content.

Good luck
K<o>