Files 404 Error Googlebot can't find

[joejoe0905]

Hi All,

Just starting out a website over at http://www.vtfy.com and submitted the URL and sitemap to google for indexing.
I just checked the progress and it shows that (15)URL's weren't found with a 404 Error.

I checked each one manually by selecting each link and all the URL's are valid.

Does anyone know what could cause this error, or better yet has anyone ever encountered this before? Thanks in advance for any help you guys can provide !!!


Here's a list of the URL's the gogglebot received a 404 error on:

http://vtfy.com/component/option,com_contact/Itemid,3/ 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/component/option,com...,lostPassword/ 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/component/option,com...task,register/ 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/component/option,com...0.3/no_html,1/ 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/component/option,com...PML/no_html,1/ 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/component/option,com....91/no_html,1/ 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/component/option,com...1.0/no_html,1/ 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/component/option,com...2.0/no_html,1/ 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/news/latest/show-desktop.html 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/video-tutorials/windows-xp/ 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/video-tutorials/wind...-in-winxp.html 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/video-tutorials/wind...s-taskbar.html 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/video-tutorials/wind...reensaver.html 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/video-tutorials/wind...documents.html 404 (Not found) [?] Mar 12, 2007
http://vtfy.com/video-tutorials/wind...-in-winxp.html

[activeco]

Your domain is just nine days old and you probably allowed access to Googlebot during testing phase which produced the description of your home page to look like this: "Joomla - the dynamic portal engine and content management system.".
It is also that the bot(s) tried the other pages too, which were not ready at Mar/12 yet.
A temporary downtime on the server is not excluded too.

The only 404 I could find from some of your links are the ones that don't return real 404 at all.
So, all those files are considered as duplicate.

e.g. the headers:
GET /winxp/recent-documents.jpg HTTP/1.1
or
GET /winxp/defragmenting.jpg HTTP/1.1

return:
HTTP/1.x 404 OK

It should be: HTTP/1.x 404 Not Found

You tried a custom 404, but made it in a wrong way.
Be careful, all 'non-existent' requests producing the same 'OK' page mean a lot of duplicate content.

You have also disallowed components/ in your robots.txt file, which was probably allowed to bots earlier.

[Kzajko]

I have a similar situation. Google caught a few pages when my website was being built. I didnt pay attention but after 1 month I still see it says that two pages are 404(Not found).

These pages do not exist 4 weeks or more and there is no links pointing to them from my website. Is it Google's mistake? I though their system would correct it but it still shows up. Is it something to be done to change it?

[incrediblehelp]

Originally Posted by Kzajko

These pages do not exist 4 weeks or more and there is no links pointing to them from my website. Is it Google's mistake?

I dont understand, shouldnt Google think they are dead then?

[joejoe0905]

Thanks for the help guys, I actually found why googlebot could not find the pages I mentioned above. It has something to do with opensef and the .htacess file, i used the .htacess file that was recommended at the opensef forums and googlebot is now able to see the urls which gave the the error before

[Kzajko]

Originally Posted by incrediblehelp

I dont understand, shouldnt Google think they are dead then?

Yes, it should but still says they are not found. These pages existed a few days when webside was being built 1.5 month ago and thats all.There is no links pointing to pages Google is telling me about.

My guess is that their web master tools are quite new and not everything works as should.

Chris.

[incrediblehelp]

Originally Posted by Kzajko

Yes, it should but still says they are not found.

Right I just said that.

They should be dead and they ARE still being found? Is that what you mean?

[Kzajko]

well, that is my point. They are death but somehow Google sees those links and It is wrong because they do not exist.

The same is with my another website. Since 2 weeks Google found 15 links broken. It was because i moved to another hosting provider. After i fixed them Google crawled my website at least 2-3 times but i still see the same message about 404(not found) links

The interesting thing is that this does not affect my position in Google search. It actually keeps going up and i recently made it to top ten.

Thus as i said before my guess is that those webmaster tools are recently introduced and Google keeps working on them.

[incrediblehelp]

Yup well Google has said they sometimes leave dead files (or 404 files) in the index. They will come back and check if they are back live sometime later. Of course on the other hand I have seen them remove 404 files very quickly.

All in all you are right, it is funky. Not sure why they decide to keep some in the index and remove others. Must have to do with the pages past "worth" or "popularity".