So lame, can this bot be prevented?

[Niko Holopainen]

Hello everyone, for the first time I encountered a bot in our site that I really don't like if it's doing what I'm thinking it's doing.

Website_Downloader_http://www.duelta.com

I'm not sure but I think it's a program that downloads the entire site. I know that it's impossible to prevent the copying of materials but copying our entire website seems a little lame.

Does anyone have any experiences with this and possible ways of prevention? I'm somehow inclined to believe that it wouldn't obey robots.txt even if included there.

Yours truly,

[eightfifteen]

I think you can specify thet in your robots.txt file.

Something like:

User-agent: duelta
Disallow: /

[paulhiles]

Niko, when I type in duelta.com in my browser, I'm automatically re-directed to Lomasake.fi
I think I'd be tempted to do a lookup on the domain name.

Update: I just checked that.. and that's your domain name... doh! :o)

Can you post the extract from your logfiles which gives the user-agent string for the bot? is it simply called Website_Downloader? most legitimate bots will carry a public page address giving further info on its purpose.

[eightfifteen]

DOH for me as well. What bot is it?

[paulhiles]

I have a feeling that rather than a bot, it's actually someone with a program called Website Downloader. I'm guessing any site they like the look of they'll just enter the URL, click submit, and Bam, all your links, graphics, etc are downloaded in a trice!

Not a great deal you can do to defend yourself against this. If you're seeing repeat visits from the same IP you could obviously block this user, but no completely satisfactory solution.

[Niko Holopainen]

Thanks for your replies, I'll ask our programmer to provide the log details.

I'm inclined to believe that this is an actual user using a website downloader program. Also, checking up on ip:s that have made such amounts of hits and pages I believe it's a cable service provider that has a dynamic ip. This would be annoying. Hmph.

Anyways, I'll post to this thread as soon as I'm wiser and thanks all for your support.

Regards,