disproportionate hits from unknown robot

tvirden · #1 (**permalink**) 04-04-2005, 10:40 AM

We usually get between 100 and 300 hits a night from Google and MSN spiders and 1 or 2 from unknown robots. Last night we got 74000 hits from an unknown robot identified by crawl. They downloaded 1.02 GB of data - MSN downloaded 7 MB. Any idea what this is from?

Thanks,
Tom[/quote]

jestep · #2 (**permalink**) 04-04-2005, 11:26 AM

The next thing you should expect is 1.02 GB of spam coming to any email listed on your website.

Have you done any type of website checking, link, validation, SEO tools, etc.

Otherwise it is probably someone's spambot that went crazy on your site.

greeneagle · #3 (**permalink**) 04-05-2005, 03:06 AM

"Crawl" has been extra active this month so far. Here is a decent thread on it: http://www.webmasterworld.com/forum11/2453-2-10.htm .

It very well may be a second bot used by one of the big boys used to validat a main bot and/or check for spam.

They have been with us for years and never presented a problem we know of.

They do act just like one of the major bots though.

Ken