Quote:
Originally Posted by deepsand
There is no way to hide from scrapers without hiding from SEs as well. Neither needs a site map in order to perform their task(s).
|
Is that because they will spoof their ip addresses to look like the SE spiders IP addresses?
I have also tried limiting the number of allowed connections to my server, so that no one IP can have more than a set number of connections going without getting blocked. As I was told that the SE do not make so many connecions, that that is only scrapers and it does not appear to have hampered the legitimate bots doin that.
But these sites, I do not think they are directly spidering my site, I know they can, but it would be alot less easy for them than just having the urls given to them in the sitemap, which can simply be uploaded to their database for use in this link hijacking, without having spidered anything.
I previously reported a couple of these sites to Google and I have noticed they have been removed, but I dont want to spend my days reporting stuff like that. I just want to secure my site so it isnt an issue and I can get on with my work.