Every time I run the Google webmaster tools and I check for page errors with the Web Crawl, Google repeatedly lists the same seven pages and subfolders that haven't existed on our website for at least two years.



These are the 404 error pages it lists (removed 'http://www.' to prevent links):

1. endoscopy.com/ess01/sx030002.htm
2. endoscopy.com/ind/choosing_a_borescope.htm
3. endoscopy.com/index.html
4. endoscopy.com/med/index.htm
5. endoscopy.com/products/ess_ent-p4.htm
6. endoscopy.com/products/forceps_graspers_retrievers.htm
7. endoscopy.com/products/medcartscabinets/

Is there any way I can get Google to stop trying to index these pages/folders? I had tried adding them to the robots.txt with no luck. As I said, they haven't existed on our webhost's server for at least two years, maybe more.

Any ideas/links I can try to use to correct this? It's getting quite frustrating!

__________________
~Mark G.
Graphic Designer - Endoscopy Support Services, Inc.

Most likely there are still links to these pages somewhere. As long as someone links to the page, and the page returns a 404 message, Google will try to index the page. 404 messages do not indicate that the page is gone. Instead, it indicates "the document is not currently accessible. However, keep trying, because it might come back." To indicate that the document is gone permanently, your server would need to return a 410 code. Unfortunately, most webmasters do not know how to return a 410 error message. You would put the following line in your .htaccess file:

Redirect 410 /ess01/sx030002.htm

(Repeating for as many pages as are Gone.)

However, if you have links to these old pages, you may not want to waste the link juice. In most cases, it would be preferable to use a 301 redirect to send the traffic to another appropriate page on your server. This preserves page rank, and will also keep any visitors that follow the outdated link.

Redirect 301 /old/page.htm http:// site.tld/new/page.htm

__________________
The best way to learn anything, is to question everything.

Thanks Wige.

Since my hosting provider kinda sucks, I've contacted them about either providing me with the ability to change the .htaccess file or I will give them the instructions that you gave me and provide them with alternative links to redirect those to other pages on the site. If you're right about them possibly being viable links, it would probably be better to redirect them rather than block them.

Question:

One of the "problem pages" is the index page - the old page was "index.html" and the new/current page is "index.htm".

Is there some way to add a line or two of code in the .htaccess file so that whether the page is index.htm or .html, browsers will recognize them as one and the same? So if down the road it was "accidentally" changed to index.html, those with links to index.htm won't get a 404 error and vice-versus?

Another question:

Can I use Google tools or some other method to "backtrack" what sites have those links to those pages? I've never done it before and I don't know how or if it's possible. There's one of them that I don't even know what it originally was (therefore, I don't know where to redirect them to).

__________________
~Mark G.
Graphic Designer - Endoscopy Support Services, Inc.

As far as index.html to index.htm, the best you would be able to do is redirect one to the other. You do not want the browser to be able to access the page at both versions, as this would cause search engines to see the pages as duplicated. Generally, I prefer to rewrite all possible index.ext files to the root of the folder (so folder/index.ext always converts to folder/). Your server should automatically serve index.html or index.htm or index.something depending in which extension you have. Assuming that is the case, you would use the following rule to force browsers to the root:

RedirectMatch 301 ^(.*)/index\.(.*)$ http://url.tld$1/

This will match any folder structure, and remove index.ext from the end, redirecting the user and spider to the directory root. For best results, make sure your internal links point to the directory root.

Also, please note that if you will be passing query strings (I am assuming this is not the case since the pages are .html) this method will not work.

__________________
The best way to learn anything, is to question everything.

Okay, they got me FTP access. In "my" directory, I see a BAK folder, data folder, and a www folder. The only other things there are two text files for our SSL certs and a zipped file of our shopping cart software. The BAK folder has a copy of all our site images, the data folder holds our database files, and the www folder is the same list of files/folders that I have access through FrontPage - basically, our website files (the www is our "root").

Nowhere did I see an .htaccess file. Is it possible to run a website without the file, or is it that I don't have access to where the file is?

Also, can a web server have a single .htaccess file that runs multiple domains/websites or do I have a specific .htaccess file for our website on that server?

__________________
~Mark G.
Graphic Designer - Endoscopy Support Services, Inc.

Take this advice very suspiciously, htaccess is not something to be trifled with and I am not a master of its full potential.

However, usually I create or see them created in the /www/ path.

If you have root access, unlikely if on a shared versus dedicated server, you can have one at root and if you like, one in each (IP) domain also - they cascade down the path iirc, so only do as little as you need to, especially in areas such as 301 redirects.

Wige is a whiz on this kind of thing so I will shaddup now.

__________________

Denver SEO, Blog, and Web Design Consultants

It's possible that your permissions do not allow you to see or modify the .htacces.

It's more likely that your FTP program is filtering the files you see. In your setup (with the FTP program) you should be able to experiment with filters that will allow you see the .htaccess.

In cuteFTP I have to choose to "enable filters" and then tell cuteFTP to let the server filters do the filtering, not cuteFTP. That's usually enough.

__________________
Jade Burnside, Ahead of the Web
What good is your web site if no one can find it?
SEO & Optimized Web Site Design