Hello everyone, for the first time I encountered a bot in our site that I really don't like if it's doing what I'm thinking it's doing.
I'm not sure but I think it's a program that downloads the entire site. I know that it's impossible to prevent the copying of materials but copying our entire website seems a little lame.
Does anyone have any experiences with this and possible ways of prevention? I'm somehow inclined to believe that it wouldn't obey robots.txt even if included there.