Tag: stomps

Craigslist Stomps Out Spiders

A tweak to the craigslist.org robots.txt file now bans all spidering, including that done by Google or Yahoo, from taking…