Quote:
Originally Posted by chandrika
|
That was a good bit of information.
If you're doing the "Ban" or "Disallow" to all those types of USER_AGENT connections I'll give a try to reverse it and only allow "Human" and the big 3 search engines.
The concept is sound but only time will tell. Drawback is that you have to create a new Disallow for any new bad bot so it might be easier to just allow the good search engine spiders, content copyright bots, plagiarism bots etc.
- Do you have a method of tracking bots attempting to copy your site or at least your articles section?
- Wound you agree to share your results in about 30 days?
I feel plagiarism, leeching, scraping confuses more consumers and directly affects valid storefront owners. I learned by the "School of Hard Knocks" by posting on wordpress some articles I wrote for my site. I will never do that again.