Non Existing Pages Being Indexed

[melchapman]

Hello All...

Recently my client transferred the domain Frankel Realty Group | Ponte Vedra and Jacksonville Real Estate from one hosting company (Advanced Access) to another (Real Estate Webmasters). The new site has all new content and page organization.

But for some reason... the Search Engines are continuing to index non-existent pages which are the pages from the old site.

Will it just take some time to get those pages completely removed? But the funny thing is... is when I run an indexation report (I use Web CEO) it shows that the search engines are adding new pages that existed on the old site.

Is there something that I can do... or again... will it just take time to get them all removed? Also... if they are still indexing these old site pages, could it affect the rankings?

Thanks so much in advance.

Melissa

[incrediblehelp]

First question is did you let the search engines know the website moved with a 301 redirect?

[crankydave]

Check to make sure the DNS is set up correctly.

If they're continuing index the old pages then they still are on the server and the DNS is not sending them to the new one.

Dave

[doj464]

If you don't want those old pages indexed, add those pages to your robots.txt file, with "noindex nofollow" command

The Web Robots Pages

[Noise-Barrier]

i had a weird issue a while back where a page was coming up in the serps for vanity search on my name that had been deleted over a year before. the page hadn't been in the serps for months after i deleted it, but then out of no where it popped back up in the serps, on the first page. luckily it went away again quickly, might've just been a glitch or something. though it does take the se's a little while to figure these things out.

[hyperdog]

You should follow incrediblehelp's suggestion to 301 redirect. Don't let your site drop from a 400 page site to a one page site in a search engine's eyes just because all of the old pages cannot be found. It can take time to get large sites reindexed.

You will probably want to 301 redirect the old page locations to the new ones. It's a server configuration issue. That way, visitors that have bookmarked - or those clicking on an older serp listing - are sent to the correct location. Search engines will eventually pick up on the new page locations. At least some of your link power to those old pages will be transferred(I think people are still debating how much, though). It sure helps the robots find those new pages, too.

We compiled some other tips into a blog posting last year:
Web design followup: What to do after the big site launch

[G[dot]com]

One question: how long means "recently"? Days ago? Weeks ago? A month ago?... If it´s days, I would not worry that much yet if it´s a month or more, it´s to be worried about.

Beyond that, yes, you should consider the 301 redirect advise mentioned above.

Cheers!

[datetopia]

Sometimes changes show up in search engine results much later. Any DNS propagation issue should get fixed in couple of days. You should wait a couple of weeks to make sure your updates are crawled and this is reflected in the results.

[bosmolskate]

This occurred with my old site. It takes the search engines a few months to update and make the changes. It took approximately 2 months for Google to remove my old site URL's from its SERP and now only my current pages appear. Try submitting a new site map to all the search engines that you have this problem with. This will help fix the problem.

[moredial]

It sounds like you did not do any housekeeping on your old hosting.

Because of the way the database index works at search engines, they use IP address as your 'ID' with your domain name and content being only secondary indexes.

If your old hosting still has your domain in the domain tree, then the search engines can continue to index that content - remember the http:// IP address/ ~username/ etc that can be used to access content before the DNS is moved? - if you could access the content then so can the search engines.

If the old site is still hosted there, it will continue to be found until such time as the host no longer responds.

The worst happens when the host starts to respond with a 404 response and then the old cache turns into a 'ghost' which can live on until the search engines cleans out its database - needs to replace the disk containing the database.

You need to ask your old host to give a gone response for all your old content. This is much better than a 301 response (OK in the short term until new host is discovered) as so many search engines keep the old url but just cache the new content on the old url rather than the new url, i.e. they treat a 301 response the same as a 302 response.

I hope the above helps get your site found on the new hosting.

[wige]

Putting a 410 Gone response on your old host is probably not the best solution, as this tells the search engine that this server is still the authoritative server for that domain name.

To prevent overburdening the DNS system with queries, most search engine spiders divert from the spec by caching DNS mappings. This cache is typically maintained for a set period (rechecked every month or so) or until the destination server stops responding to requests for that domain name. Typically, if the server responds with an error 400 Bad Request (this should happen very soon after you close your account - if you don't trust your host, you can always try to hack the response codes, but it may be risky), the first thing the spider will do is query the DNS system to find out if the domain name has moved.

Hacking server response codes:

The following entry in your root level .htaccess should force your old server to respond to queries looking for your home page with a bad request message:

Redirect 400 /

If that fails, you can always upload a php file containing the following:

PHP Code:

<?php
header('HTTP/1.0 400 Bad Request');
?>
<h1>400 Bad Request</h1>
<p>The hostname specified is no longer available on this server.</p>
<hr>
<p><i>Apache</i></p>

Also, again to repeat what has already been said, make absolutely sure that you are 301 redirecting the individual removed pages to the new locations on the new server. If a search engine sees a 404, especially after a move, according to spec this is a temporary condition. The search engines will keep looking for the content for up to a year. Longer if there are valid links to those URLs. 404 literally means that a resource is unavailable for an unknown reason. 410 should be used if the content is gone and will never return and there is no forwarding address. 301 should be used if the content is simply at a new URL.