Mysterious entries in robots.txt reported by Webmaster Tools

[wedshop_master]

Over the last year I have enjoyed watching all of my 'web crawl errors' disappear from the Webmaster Tools 'overview screen'.

A few days ago I logged in to find that I had gone from having 1 page not found, and 1 page restricted by robots.txt,
to 4 pages not found and 90 pages restricted by robots.txt, and bizarrely none of the entries have ever been in my robots.txt file,
and none of the 'pages not found' are actually pages that have ever existed!

It doesn't seem as though any of this is damaging, but it did make me think that the site had been compromised somehow.
All of the pages restricted in robots.txt are mobile phone related (nothing to do with my bridal themed site of course).

Here is a screen grab of the first few entries:

[activeco]

It seems you wrongly implemented custom 404 code.
Chack this page: How to Set Up a Custom 404 File Not Found Page on your Website (thesitewizard.com)

[wige]

My guess, you have been hacked. I have seen a few attacks very similar to what you are showing. Have your hosting company pull your FTP logs and check for any anomalous activity. Also, go through every single folder on your site and grab the .htaccess files. Look for mod_rewrite RedirectRules that have been added, as that is the most frequent method used in this type of attack - it allows the attacker to hide a single file somewhere you are not likely to check and then create internal redirects to that file.

Also, check the lastmod date of the robots.txt file, and search for all files on your server that have a lastmod within 6 hours of that time, as those files may have also been altered. You need to do this through a command line (SSH connection) not through FTP, which may change the date. If you don't have secure telnet access to the server, have your hosting company compile the listing.

I did notice that these entries do not show up if I look at your robots.txt file. Have you uploaded a clean file, or are these entries not actually there? If the entries are not there, the attacker may be using a mod_rewrite rule to serve Google with a different robots.txt file. The only way to confirm this would be to sign in to your Webmaster Tools account and have Google analyze the robots.txt file.

[wedshop_master]

Originally Posted by activeco

It seems you wrongly implemented custom 404 code.
Chack this page: How to Set Up a Custom 404 File Not Found Page on your Website (thesitewizard.com)

Thanks activeco.

Non-existent pages are returning 'HTTP Status Code: HTTP/1.1 404'.

So the 404 code appears to be doing the job.

[wedshop_master]

Originally Posted by wige

My guess, you have been hacked. I have seen a few attacks very similar to what you are showing. Have your hosting company pull your FTP logs and check for any anomalous activity. Also, go through every single folder on your site and grab the .htaccess files. Look for mod_rewrite RedirectRules that have been added, as that is the most frequent method used in this type of attack - it allows the attacker to hide a single file somewhere you are not likely to check and then create internal redirects to that file.

Also, check the lastmod date of the robots.txt file, and search for all files on your server that have a lastmod within 6 hours of that time, as those files may have also been altered. You need to do this through a command line (SSH connection) not through FTP, which may change the date. If you don't have secure telnet access to the server, have your hosting company compile the listing.

I did notice that these entries do not show up if I look at your robots.txt file. Have you uploaded a clean file, or are these entries not actually there? If the entries are not there, the attacker may be using a mod_rewrite rule to serve Google with a different robots.txt file. The only way to confirm this would be to sign in to your Webmaster Tools account and have Google analyze the robots.txt file.

Thanks wige, this has been very useful advice because although I didn't find anything suspicious,
I wasn't aware of the role the .htaccess file plays, and discovered it had a double entry for my 404 CGI script.

My robots.txt file is clean, and as far as I know always has been, but Google Webmaster Tools is now reporting 236 restricted URLs, again all related to mobile phone ringtones.
All of the mystery URLs are within a folder called 'ringtones' in my 'images' folder which is restricted in my robots.txt.

[activeco]

Originally Posted by wedshop_master

Thanks activeco.

Non-existent pages are returning 'HTTP Status Code: HTTP/1.1 404'.

So the 404 code appears to be doing the job.

You were right. My wrong thinking.
However, there are many links to the illegal pages: "theweddingshop.co.uk/images/ringtones/ - Google Search
My thought was about returning wrong 404 for them, but robots.txt didn't make much sense in that scenario.

Some sort of hacking is possible as no page returned was cached: site:theweddingshop.co.uk/images/ringtones/ - Google Search, but again, it could be due to the robots.txt exclusion (?).

P.S. I just found out that you indeed have /images/ excluded in robots.txt, so Google was technically right. The chance is good that competition is trying to hurt your site by bad linking.

[Jean-Luc]

As activeco wrote, your robots.txt disallows access to these URL's. Google found these URL's in other websites where they had been posted by a spammer.

It is likely that the spammer did that, because he thought he was able to control these pages on your website. It looks like the spammer is wrong and that his efforts here are pointless. Maybe he is using hacking tools that are buggy. Maybe he is stupid and he was not aware of your present robots.txt file.

You should doublecheck that each and every file on your server is clean. This situation is not going to hurt you in any way as long as the spammer has no control on your server.

Jean-Luc

[wedshop_master]

Originally Posted by Jean-Luc

As activeco wrote, your robots.txt disallows access to these URL's. Google found these URL's in other websites where they had been posted by a spammer.

It is likely that the spammer did that, because he thought he was able to control these pages on your website. It looks like the spammer is wrong and that his efforts here are pointless. Maybe he is using hacking tools that are buggy. Maybe he is stupid and he was not aware of your present robots.txt file.

You should doublecheck that each and every file on your server is clean. This situation is not going to hurt you in any way as long as the spammer has no control on your server.

Jean-Luc

Thanks Jean-Luc

There are loads of horrid spammy pages out there with dead links to my site now (I've used Webmaster Tools to remove the ringtone directory and any of its subdirectories and files).
I really hope (and would assume) that this won't damage my ranking because obviously anyone can easily do this?

[wedshop_master]

Originally Posted by activeco

The chance is good that competition is trying to hurt your site by bad linking.

Thanks activeco, I'm beginning to wonder if this is the case because we've just started to get decent top 5 rankings for our important searches. Maybe someone has hired a 'web hit man'?