spiders!

[joesnow]

Hi

Hope I'm in the right forum for this!!

Just noticed that the Inktomi spider is hitting our site (http://www.golfgroups.co.uk) then buggering off pretty sharpish, all site visits are only for 00.00 seconds, apart form once or twice, when the spider has visited for around 9 minutes, but only visited 2 pages!

Anyone have any ideas whats going on with this spider or my site???

Thanks a lot!
Joe.

[KeithO]

sounds lost to me. do you have a site map?

[joesnow]

yeah, got one at http://www.golfgroups.co.uk/sitemap.html

So that should be cool, all straight links etc..the only way to get to it though is by click on an .asp
include file, wonder if its having trouble finding it?

Joe.

[joesnow]

anyone else got any ideas out there? stumped me this one...

[ctabuk]

No, according to www.SiteReportCard.com it has a slow download time, but I did not find that when I looked direct, no broken links, nothing wrong at all. Just one of those things I guess. Do we get free holidays now?

[wrmineo]

A few guesses:

First, the slow download time is likely to be the same cause for the MSN bot having issues; javascript. It doesn't affect most users or the actual download for most, but bots aren't crazy about it for some reason. I run js on most of my sites without real concern or issue, but I've noticed that the sitereportcard and others seem to hang up on it ...

Also, I couldn't find a robots.txt file; MSN could be "excusing itself" when it doesn't find permission to be there.

Finally, validate your code.

Good luck and happy rankings!

[joesnow]

Hi guys

Thanks for your responses!

Anyone know what the best robot.txt is to use at all?

Also when you day optimise your code, got any more info? Or a URL where I can look at this?

Cheers
Joe.

[wrmineo]

Check out some of Google's Webmaster Guidelines and for a good robots reference, even the Big G points to this Robot Reference

Can you rephrase or give more details on

Also when you day optimise your code, got any more info? Or a URL where I can look at this?

as I'm not fully sure what you're asking here?

[joesnow]

Oh, cool, thanks I'll take a look.

Sorry, I meant validate my code, as in, whats the setup?

Is there somewhere that gives you an idea of the best practices to do this?

I have a coder who works with the site so they could take a look for me...

Cheers
Joe.

[wrmineo]

Personally, I always go with the W3C Validator; they are the source all others benchmark from IMO! I'd start there ... good luck!