The robots.txt file should not make a difference. Although, you should probably have one. Most of the SE guidelines "suggest" it and no telling what Slurp does if it does not find one. But, I repeat ... it should not make a difference in having your site crawled.
It does appear that Slurp is pretty much ignoring you. What you saw in the Yahoo index previously, was more than probably old Google results. You just happened to catch the results at the right time. They do no use them all the time, and at the end of the month it will be history.
I think you will need to work on getting some backlinks to spur Slurp on a little more. A lot of people have NOT been crawled by Slurp with any reasonable amount of frequency, so you are not alone on this. The links should definitely help improve that though.
Also, since Slurp has crawled your home page twice and I can see that there is only 2 bytes difference between those crawls -- it may appear to Slurp that your Home Page does not change. Perhaps adding a couple of K to the page (or losing about as much) will pique it's interest more. I do this from time to time, even though not much changes ... hehehe. And with bots in general, they think something new has been added. It usually sparks (googlebot anyway) into crawling a little deeper.
You also seem to have got a response out of Slurp rather quickly anyway. Usually it has taken some people about 8 or 9 weeks to even see Slurp at their front door. So ... in that respect you are doing okay. I would keep working on those links and make sure it does not lose interest.
|
Contact Us
Forum Rules
Search
Archive
Linear Mode
