Googlebot only visting index.asp and robots.txt only

[pbatson]

Hi,

am fairly new to seo so this may be a stupid question but have been looking at the logfiles for a site that I have just submitted to google and it shows googlebot visiting each morning but only visiting index.asp and robots.txt - why does it not crawl around the site - is it because its dynamic? I do have static .htm links from index.asp.

Would appreciate some advice (for a seo newbie ;-)

paul

[adore]

Is your site new?
Does the home page include text links to other sections of your website?

[minstrel]

Can you post the URL to this website, pbatson?

If not, as adore implies, if the site is new and has a low PR, Google may not yet have spidered all the pages but if it found the index it should eventually find the rest. Make sure that the links to your dynamic pages are fairly short and double check that robots.txt file for errors while you're at it. You can also check the URL in a Google search and see if it gives you the option of "more pages from this site" - if so, it means the other pages are indexed but Google isn't displaying them all in a normal search.

[pbatson]

Thanks for taking the time to reply,

Yes adore - site is quite new - only submitted to major se and directories about 3 weeks ago. Text links hardcoded in html to main boilerplate pages + text links dynamically generated from db to product pages.

minstrel - URL is http://www.funkycrocodile.com

links to product pages are fairly short - eg

defaultProducts.asp?Manufacturer=Baby Gap

have took your advice searching on google but does not give option of "more pages from this site" so presume this implies other pages are not indexed.

On the subjuect of the robots.txt file - I do not have one - I understood they are really only if there are pages you do not want indexed?

Thanks agn

Paul

[Mel]

Hi Pbatson
I find only your home page in googles index, but your site seems to spider well, and the links to your other pages are found.

It may well be that you will just have to be patient, and in the meantime you might want to look into getting your server to return file dates so that it can respond to Googles If modified since GETs, since Google likes to spider pages that respond properly to IMS requests.

I note that you set a cookie when your page is visited, just make sure that it is not mandatory as spiders don't accept cookies.

[pbatson]

Thanks mel

I will approach my host and see what they say about rerturning the file dates.

As for the cookie i don't think its mandatory....

(probably shouldnt be in a thread on google but here goes)
I only set it to allow the user to keep the selections they've made eg. can select boys then select size 0-3m and it will remember the boys is still selected. So if a spider visited the page it should just visit each page individually eg all makes, then boys, then girls, then unisex, then manufacturer adams, etc. just going through each link seperately. This would mean some products are repeated on each page though so could influence googlebot for repeat content maybe?

Thanks

Paul

[minstrel]

re: robots.txt file

No, you do not need one, but as you have seen googlebot looks for one and it doesn't hurt. Create one and upload it to your default directory with the following contents:

User-agent: *
Disallow:

Just that, exactly as it is typed here. This says to all spiders, "please index everything on this site" (or if you want to be a purist, "do not exclude anything on this site from indexing").

[Mel]

Since you are running on an IIS server (and it looks like .NET too) I think you will find that the retun of the file date has to be be programmed pagewise instead of serverwise.

[Woofer]

I put up a couple new static sites a few weeks ago, along with links for Googlebot to follow, and the bot has only came in and crawled their robots and index pages, which have been indexed in Google.

Until recently, similar new sites had often been getting fully crawled within a couple weeks, and fully indexed a few days later.

So it looks to me like Googlebot is taking a bit of a break in regards to crawling new sites.

[pbatson]

will do that minstrel.Thanks.

Mel, I'll have a look at the books (still on a learning curve with ASP but getting there...) so thanks for the advice.

Woofer - I'm glad i'm not alone... ;-)

paul.

[mediahound]

May I suggest SpyderTrax

Check it out at www.DarrinWard.com

This script will easily allow you to track the robots as they crawl your site.

Let me know how you like it

[JollyGoodFellow]

Since 3 weeks I notice Google only crawls my homepage, while previously it crawled the whole site.

Further more, it looks to me that google has uploaded back-ups a couple of times the last few weeks. I notice this because my homepage keeps changing to the "old" contents when I click on the "Cached" link in Google.

Like a yoyo:
"old cache ,new cache........" every couple of days
and the rest of my site isn't crawled anymore.

Yeah, I think Google is testing .......(something).....and they have trouble fine tuning it

www.surferquest.com

[mediahound]

Did anyone try the spydertrax software/script I mentioned above?

I'm working with the creator on a new project everyone should enjoy!

Regards...

[pbatson]

sorry MH haven't had a chance yet - did look at the url but tend to be just looking at the logs at the moment (it's a new site so wanted to see what pages/products people are looking at and therefore see which robots have crawled it then).

Paul

[mediahound]

I updated a domain last night, you can visit www.spidertracks.com most likely by tomorrow to get the program.

Though it's still available at www.darrinward.com

Your going to love it!

Its really a good spider tracking program...

It also now works on php pages for those of you out there that have used it in the past, or are currently using it... It also tracks the 'new' yahoo slurp spider.

We're working on a centralized site that can do the tracking for you, without installing the script on your server.

Post here if you've used it, or have any feedback on it, or if you need help installing, etc..

Have fun out there,

Jarred