Thanks DPA guy for the quick response and great advice. Here's the problem. My site is already search engine optimized(or at least to the best of my limited knowledge). All the other crawlers have no problem crawling my site and it has over 500 pages listed in google. Since my site is also a usefull community based site there is lots of IBLs to it from sites within it's topic. I'm wondering if Ask Jeeves purposely sets their robots to only spider 1 page (you pay for one you get one) of your site for paying members. After all if your a big enough sucker to pay once you might as well pay by the page. I'm curious, does anyone have any theories on this?
|