Competition spiders?

**brian.mark** · #1 (**permalink**) 10-27-2004, 02:13 AM

We've heard from one of our vendors that a competitor is using a spider to gather pricing information and automatically adjust their prices to beat ours and the other competitors. I have a couple of questions on this.

1) Is this legal? I mean, that would cost us money in bandwidth, CPU time (we've been considering upgrading servers because of peak times, etc.), as well as causing some bottlenecks at other places on the web.

2) How would we go about catching them? I've looked for unusual user agents, and I don't really see much. I know it is a simple matter to cloak a bot to look like a legit browser, so how would I identify it? I have come across a few IP's that are using more bandwidth than others, but they don't seem to resolve to anywhere close to this competitor's IP block.

Any suggestions are welcome.

Brian.

flood6 · #2 (**permalink**) 10-27-2004, 04:08 AM

I don't think it's illegal for them to use bots to scrape your site unless unless you specify somewhere that you don't allow that sort of thing.

I agree that identifying a bot like that would be your biggest challenge. IP will most likely not be a big help; they could run the bot from their web host, their office, their home, etc.

If the bot obeys your robots.txt, that might be a start; look for agents that look like humans but pull your robots.txt. But be careful, looking at robots.txt's has become something of a hobby of mine. I'll check it during a site review, or considering a link exchange partners, sometimes I'm just curious; I imagine other developers also do this from time to time.

If this bot doesn't grab your robots.txt or intentionally disobeys it, you can set up a bad bot trap such as the one described here.

Good luck.

benc007 · #3 (**permalink**) 10-27-2004, 08:17 PM

You can block IPs that have search a significant number of pages on your site. I wouldn't recommend this as the competitor can use multiple proxy IPs. Any ideas on this topic? I would like to know more about this as well.

Thanks,
benc007@excite.com

Keimos · #4 (**permalink**) 10-27-2004, 08:35 PM

Hi Brian,

They could simply log onto your site and cut the prices anyway. They have simply automated the process. Its been happening in retail for years.

If you have a legitimate site and a good customer base and offer integrity then buyers will stick with you. Basically we all shop to penny pinch but there is a difference between saving $1 or $5 or $10 etc.

You know what your profit margin is and should have a very good idea of what the general price is for your products. If they are undercutting too much they will go out of business anyway.

Annoying as it seems live with and get on. It is legal, as far as I know, because you can sell anything for any price. Whether you make a profit is another matter.

Identifying their IP is a waste of time, so go and have a few cold beers and forget about it for a while.

The one suggestion is to look at your pages that are indexed well and exclude robots from the other pages.

To me not a good suggestion but could possibly work by allowing all the category pages to be linked but exclude all the product pages. I am just having a guess here so think about it before trying anything.

Firwefly · #5 (**permalink**) 10-27-2004, 08:35 PM

benc007 is right: it is legal (depending maybe on place of jurisdiction). It is also good business sense. I've always used every means possible to gain competitive data and always been successful ;)

Andilinks · #6 (**permalink**) 10-27-2004, 08:54 PM

I specifically disallow unapproved bots on my site, my notice is here near the bottom of the page:

http://www.andilinks.com/pol.htm

Identifying the bots for banning is a little trickier but you'll find that even when they try to disguise themselves as browsers they usually don't download your images or css files etc., so any domain that is downloading significant bandwith with too few hits per kb can be safely assumed to be a bot. Say if your typical page is 30K with ten extra files called per page and an IP is downloading 90K with just 3 hits it will definitely be a bot. You can then ban the IP with .htaccess.

I find many are just one-time visits but if your competitor is scanning for prices they will probably be using a static IP or the same block of IP's every time.

Often I will deny the entire range, omitting the last set of 255 of the IP on the .htaccess.

That is: Deny from 255.255.255.

Instead of: Deny from 255.255.255.255

Andi

abell · #7 (**permalink**) 10-28-2004, 03:22 PM

That is really quite a bad idea - firstly if someone wanys to copy content off your site blocking the IP will have little effect. They will get another IP, use a different ISP etc.

Secondly how can you be sure they are not a bot and not a legitimate search engine / user.

Thirdly blocking a whole class C means not only are blocking them but also 250+ other people who may have wanted to view your site.

Lastly I would not waste the time - better spending it on marketing etc.

BGump · #8 (**permalink**) 10-28-2004, 04:15 PM

I say find a way to redirect the bot to a mirror of your site that only that one particular bot can see (no other spiders or humans) and change your prices on this mirror site to $0.15. This way, if they are not paying attention, they'll get burned. (at least for a short while)

This is probably much easier said then done, and may even step into a gray area of the law. I dunno. But thinking about, it sounds fun. :)

Medianeer · #9 (**permalink**) 10-28-2004, 04:45 PM

Instead of prices, worry about getting more traffic than your competitor. They could be giving their products away for free, but if you get more traffic, you'll be making more money.

urknighterrent · #10 (**permalink**) 10-28-2004, 05:12 PM

I don't doubt the legality of this technique, but it's definately not cool: lazy at best, unethical at worst. I don't thing banning bots is a reasonable response, and I doubt a bot like this would accept the order anyway.

I agree with Bgump. It might take some trial and error, and you might need to take some seo risks (redirects and mirror pages, etc) but it should be a simple matter to trick a bot into selling at below cost.

It's that or... well... just deal.

MeanSEO · #11 (**permalink**) 10-28-2004, 05:21 PM

Quote:

Originally Posted by Medianeer

Instead of prices, worry about getting more traffic than your competitor. They could be giving their products away for free, but if you get more traffic, you'll be making more money.

Great topic.

Great post.

I agree traffic is king. If you get 10 times more traffic so what if you lose a few percentages who got low balled.

Be sticky, have a great site that people like better, pump in quality traffic.

Anything else you do can lose you more sales in the long run by losing SE rank or blocking legit customers...or worse yet...god forbid have people buy all of your products at the posted price of $0.15!

Creatix · #12 (**permalink**) 10-28-2004, 06:08 PM

It is all very nice to sit back and speculate if this is right or wrong. Personally, I commend the competition for being clever enough to do this. However, if I was in your shoes I would be frustrated, and want to do something effective about it.
So give him some healthy competition.

Choose a couple of products that you can affôrd not to sell for a few days.
Place an out of stick notice on your site so that no one purchases them. Then (the fun bit) lowere the price very dramatically, I mean quarter it even. If their bot picks up the price, and lowers theirs to match it, they will surely be hurt quite quickly by their own handiwork.

It would also be interesting to test how soon their bot picks up the price change, and may give you an opportuinty to narrow down to the exact IP used.

As far as legality goes... they are well within their rights, and maybe you could look at creating a similar bot to protect your interests.

Ok, so my idea is crazy... but it does sound like fun.

lwscomputing · #13 (**permalink**) 10-28-2004, 10:16 PM

I can not see it being illegal, because what is to stop them browsing manually and then using the pricing ?

Over here in UK the petrol stations do something similar. Some guarantee to be lower than any within five miles. (they charge too much anyway - but that is another hot topic)

Cheers

Denis

stephenmunday · #14 (**permalink**) 10-29-2004, 05:54 AM

I'm not very techinical, so I am not sure about whether this would work.

Essentially, it would be great if you could find a way to redirect the bot like BGump said, but send it off to your competitor's site! If this worked (unlikely I guess) it would be great watching the bot and the site get into a loop until their prices all dropped to zero...

DT99 · #15 (**permalink**) 11-05-2004, 09:46 PM

There was a big lawsuit a while ago: Ticketmaster vs Tickets.com. It was about deep-linking and prices. Here are some links* you may find interesting:

http://www.gigalaw.com/library/ticke...-08-10-p1.html
http://euro.ecom.cmu.edu/program/cou...3/pollstar.htm
http://www.lawyeringskills.com/sorki.../appbrief.html
http://www.ipcounselors.com/20000508.htm#a2
http://gigalaw.com/library/ticketmas...000-03-27.html
http://www.linksandlaw.com/linkingcases-deeplinks.htm

*All trademarks belong to their respective companies.