My site has Vanished from google???

[wige]

Originally Posted by Easywebdev

I just got through reading the .htaccess file you referred to in your third post. For crying out loud delete that crap from it, in fact delete your .htaccess altogether unless there is some specific rewrite rule you must have.

SetEnvIfNoCase User-Agent "libwww-perl/" bad_bot
SetEnvIfNoCase User-Agent "^libcurl-agent/" bad_bot
SetEnvIfNoCase User-Agent "^Java/" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/4\.0\ .*Win\ 9x\ 4\.90.*$" bad_bot

Bad bots my arse.

Just a quick comment about these specific user agents.

"libwww-perl" is the user agent for the LWP client/server library for Perl. Many security analysts recommend blocking this user agent, as the LWP library is one of the most frequently used in creating automatic attack scripts. Common uses include automating requests for known CMS pages to detect CMS systems with known vulnerabilities, as well as automating the submission of form and blog spam.

"libcurl-agent" is the user agent for the libcurl multi-protocol file transfer library. This is commonly used by e-mail harvesters and scrapers. Numerous analysis sites including Project Honeypot consider this traffic malicious.

"Java" will block Java-based browsers. This are browsers and spiders that are created in Java, not web browsers that have a Java Runtime plugin. Blocking this user agent will not adversely affect the experience of users on your site, even if you have Java content, since the plugin will use the browser's own user agent. (This is a generalization, depending on the version and implementation of the browser, JRE and applet - if you do have Java content, test this) The Java programming language supports a wide range of web crawling features that makes it easy for a malicious user to create a program to crawl sites for almost any purpose. I have seen no legitimate browsers that use "Java" at the beginning of the user agent string.

"^Mozilla/4\.0\ .*Win\ 9x\ 4\.90.*$" will block Windows 98 Internet Explorer version 4.9 requests. This is done because this user agent is frequently spoofed by bad bots. Personally, I don't block this one, because traffic on my site conforms to expected traffic levels, and when comparing server logs to Analytics, the traffic seems legit.

TIP: one way to check "legitimate" user agents vs bots is to see if the IP address shows up in your server logs and in Google Analytics. Most bad bots don't download off-server Javascripts, and so won't be detected by Google Analytics even though they show up in your server logs. Of course, it is important to note that some security and privacy software applications will also block Analytics, so this is not a perfect solution, more one tool of many.

[activeco]

Originally Posted by wige

TIP: one way to check "legitimate" user agents vs bots is to see if the IP address shows up in your server logs and in Google Analytics. Most bad bots don't download off-server Javascripts, and so won't be detected by Google Analytics even though they show up in your server logs.

Doesn't it interfere with users with disabled javascript?

[Singtho]

Hello
One of the sites I work for holidays-4youdotcom
One day on the 4th page and next day gone, this is a site with many pages and competition about accommondation in Thailand, I try to do what google whant but some days we are on page 1, 2 or 3 and the next day it's gone. I am adding pages and upload new google sitemap almost every day.

Anybody know the reaso of this???

Best Regards to You all
Singtho
Singtho WebDesign

[Webnauts]

Originally Posted by Singtho

Hello
One of the sites I work for holidays-4youdotcom
One day on the 4th page and next day gone, this is a site with many pages and competition about accommondation in Thailand, I try to do what google whant but some days we are on page 1, 2 or 3 and the next day it's gone. I am adding pages and upload new google sitemap almost every day.

Anybody know the reaso of this???

Best Regards to You all
Singtho
Singtho WebDesign

Do you realize that you are hijacking the thread? If you have questions, please start your own thread.

Thanks.

[watto]

Good call webnauts. This is my longest thread ever.......don't ruin it now.....lol

Ok, I have some GREAT news, I'M BACK!!!!!!!!

My url has been re-indexed and I have the same positions as before. I can't begin to tell you how relieved I am.

[Webnauts]

Originally Posted by watto

Good call webnauts. This is my longest thread ever.......don't ruin it now.....lol

Ok, I have some GREAT news, I'M BACK!!!!!!!!

My url has been re-indexed and I have the same positions as before. I can't begin to tell you how relieved I am.

Would you please be so kind and apologize for your mistrust in my competency and your misbehaving? I would really appreciate that.

[incrediblehelp]

You were probably never gone. Just gone on some datacenters that you were checking.

Either way, nice to see you back on the datacenter you have been checking.

[activeco]

Originally Posted by Webnauts

Would you please be so kind and apologize for your mistrust in my competency and your misbehaving?

[watto]

Thanks for everyones input in this thread.

Incredible help, I was definetely de-indexed. No doubt. I checked many different datacenters through my contacts and I was getting the same result.

I'm still learning every day more and more about SEO, and a lot of my knowledge comes directly from this website.

Anyway, webnauts, I apologies. Thanks again for your help. I highly advise using your services and will do so in the future.

watto

[Webnauts]

For the last time I insist that your htaccess had nothing to due with your canonicalization problems. If someone believes the opposite, please correct me.

Looking forward to share my success with you and your site.

Take care.

[watto]

Thanks again.

[Webnauts]

I think this will answer all left open questions about the issue discussed here:
Google Webmaster Tools Incorrectly Reporting Some 404 Errors

Enjoy!

[activeco]

Watto, what version(s) of Googlebot visit your site?

[watto]

I'm not even sure how to tell which version of googebot visit my site?

[activeco]

Check your access and error logs and find out Googlebot, something like

cat your-log-file |grep ooglebot

.
where your-log-file is your file name.

Any 'Googlebot/1.0'?

[Webnauts]

Peter did you mention in this thread what you wrote me in you email, that your GWT is now showing over 5000 pages with a 404 error?

And you do not have that dirty no good .htaccess code.

Just to make some things clear for the thread.

[watto]

That's true webnauts. Currently in my GWT I have over 5000 pages reporting a error 404 (page not found), even though I removed the code. I would like to take this opportunity to formally apologies to webnauts in relation to this thread. I know webnauts didn't take things personally.

Google really has their work cut out when trying to fix their current issues with GWT. Is anyone else experiencing faulty error reports?

watto

[activeco]

Have you found out the Googlebot version info in your logs?