Slow site problems and spiders

[thepoacher]

Hi all, I need the webproworld hive-mind once more!

Our company sites (Serenity Holidays is the company, within which there are 6 brands and 7 sites) are currently experiencing severe but intermittent problems with slowness. My own technological knowledge is pretty limited so this side of things is being dealt with by (already snowed under!) tech guys. At present there is no obvious root cause - it might be a server issue, it might be the interface between the front end sites and the in-house database system we run. The intermittent nature of the problem is only making it harder to work out as it's hard to capture exactly when it happens! We think we've got it, solve the problem and then it reoccurs...I doubt a user would be able to gauge the problem but just in case here's an example site Holidays to Cape Verde with The Cape Verde Experience.

Naturally this is having a deleterious effect on our SERPs as I assume the spiders get as exasperated as a user might do in having to navigate a slow site. Is this the case?

I realise this is probably a bit of a case of clutching at straws but if anyone has any advice that would be great

many thanks

Matt

[wige]

Yes - a slow site will have an effect on the crawl rate of your site. If a spider from a major search engine detects significant delays when accessing your content, the spider will move your site to a lower crawl priority to improve it's own performance (the time spent waiting for a response from your site could be used to get one or two pages from another server) and also to minimize its impact on your server. As a result, changes may take longer to appear in the SERPs, and new content will not be indexed as readily. Over time, this could impact page rankings.

I pulled up your home page and it did take a long time (about 20-25 seconds) to load, which is a long time considering that the page has no major apparent media files. I ran the page through YSlow, which found:
This page has 20 external JavaScript files.
This page has 3 external StyleSheets.
This page has 8 CSS background images.
Wow. Thats a lot.
I also found a hidden (0px by 0px) media file: http://interface.audiovideoweb.com/lnk/va92win15202/CapeVerde/Destination/Cape-Verde-01-Destination.wmv/play.asx. This file does not load, which may delay the overall loading of the page - and what is it for anyway?
I also found http://select.evisitanalyst.com/eva/harvester/id-image.php?id=16428&timecode=1192716021058&ref=http ://www.capeverdeexperience.com/&req=http://www.capeverdeexperience.com/docs/about_the_islands/discover_the_islands/unique_islands.aspx&size=1280X1024&col=32&java=tru e&cookie=true&section=The%20Cape%20Verde%20Islands ,%20Sal,%20Boa%20Vista%20and%20more which is another 1px x 1px file, which judging by the name is a tracking script. Again, this may be slowing the site down.

[subsystems]

Try optimizing the site by using something like Web Page Analyzer - free website optimization tool website speed test check website performance report from web site optimization

But I don't think that is your worst problem. The first time I loaded your website it took about 45 seconds on a 4.5Mbps connection. The second time it took about 8 seconds. Not sure if it's server load, bandwidth or DNS issues. Also look into compressing your HTTP with mod_gzip.

[activeco]

Just to be sure it's not your local nameserver problem (as you never know with windows servers) I would add the PTR (reverse DNS) record to resolve correctly back to your domain name.
Check DNS thoroughly as you may have more potential problems (e.g. one of your mail servers mx1.capeverdeexperience.com doesn't have ip address at all).

[BOM]

Try DNS tools, reports and Hosting tests, advanced network and domain name tools. it shows the root of the problems you're talking about.

[espmartin]

Just to add a simple speed test here:

Your site took like 30+ seconds before anything loaded. I would have left if I was not testing, and just a typical surfer...

[sunpost]

a few things to try...

check your web.config file that debug is false:
<compilation debug="false" >

Precompile the site and do not make it updatable.

Consider using HTTP Compression(IIS or 3rd party) which will make those ComponentArt controls load faster

[saxman1973]

Originally Posted by wige

Yes - a slow site will have an effect on the crawl rate of your site. If a spider from a major search engine detects significant delays when accessing your content, the spider will move your site to a lower crawl priority to improve it's own performance (the time spent waiting for a response from your site could be used to get one or two pages from another server) and also to minimize its impact on your server. As a result, changes may take longer to appear in the SERPs, and new content will not be indexed as readily. Over time, this could impact page rankings.

I pulled up your home page and it did take a long time (about 20-25 seconds) to load, which is a long time considering that the page has no major apparent media files. I ran the page through YSlow, which found:
This page has 20 external JavaScript files.
This page has 3 external StyleSheets.
This page has 8 CSS background images.
Wow. Thats a lot.
I also found a hidden (0px by 0px) media file: http://interface.audiovideoweb.com/l...n.wmv/play.asx. This file does not load, which may delay the overall loading of the page - and what is it for anyway?
I also found http://select.evisitanalyst.com/eva/...a%20and%20more which is another 1px x 1px file, which judging by the name is a tracking script. Again, this may be slowing the site down.

I agree with wige. Dude! 20 external javascript files and 3 different exteral CSS files?? I hope that there is a reason for this, but I can't think of one. I can understand wanting to keep everything all nice and neat, but seriously... 20? The first thing I want to know is how large is the flash file? It shouldn't be but 60k. It it's larger you need to optimize the images better before you import. Second, I can't say I really recognize the way you have the flash file set up, but it looks like you have a default JEPG in place which is great, but I hope one of those external .js files is loading the .swf externally, but I don't think you are (I could be wrong though). Finally try and compile those CSS files and just comment out the different sections.

On another note in reference to your rankings... you are abusing your <h> tags. You should have 1 <h1> tag and 2-3 <h2 tags tops. Then use them in the way they were meant to be used... as highlights. <h> tags are to be used like highlighting a book... only highlight the words or phrases that are of major relevance to that page. If you feel you need to use more then you have to broad of a topic for that page and should consider adding new pages. Search Engines want to see relevant pages, not a page trying to explain how the world was created on one single page. It's a multiple step process, spread it out.

I would have to say right off the top of my head that all of the external javascript files are your major concern though. Hope this helps, I wish you well - L8r - Saxman

[Arctic Fox]

My experience with a server slowdown as opposed to a site slow down was one where the owner of another site (hosted on the same server as mine) decided he was going to set up a porn website. The server couldn't handle all the traffic he was getting and everyone else whose site was hosted there slid down to a crawl.

But, I agree on all the JS and CSS; fix that anyway. You may also want to contact your host and ask them to check the bandwidth.

[sensei]

In my opinion ( based on previous experiences ) yes, having your site loading/responding slowly will cause problems with the search engine spiders and those problems will be directly reflected in the SERPs.

Secondly, a slow site is also something you don't want to show to your surfers. Whenever I come across a site that loads slowly I know it has problems with the hosting and IMHO serious businessmen do take care of hosting as it is one of the most important factors for having a successful site.

[milonic]

Loads in just a few seconds for me, did you manage to fix it or is this a location problem (we are both in the UK)?

I see that you do have some DNS problems and although these don't appear to be that bad, they will need looking at so that you can dissmiss them as the cause. If the problem only appears the first time you visit the site it's got a good chance of being DNS related.

However, you mention that it's intermittent so sounds more like a hosting/connection problem. I see an IP Address in your DNS records for 84.45.#.# - This looks like it might be connected with a company called Entanet who do web hosting. We used to use them and had this very same problem until we moved our servers accross the pond. I asked them to fix the errors at dnsstuff.com for enta dot net (who we needed to do our reverse lookups) and they said their dns servers had been fine for years and there was nothing wrong with them - dnsstuff.com said otherwise.

Anyway, my summary is that it's a hosting issue and yes, as others have said, it will affect how SE's crawl your site so it needs fixing asap.

HTH,
Andy

PS - I still use Entanet for my ADSL so that might explain why I'm seeing the site at full speed if we are on the same network.

[thepoacher]

Hi all,

some really great pointers here and at present we're working through some of them to see if it gets us anywhere! The problem is really frustrating - and mainly because of its intermittent nature. It becomes very hard to isolate the problem as and when it happens, and to locate the root cause of it. It also happens on our Gambia site at The Gambia Experience The UK specialist tour operator to The Gambia.

As far as page loading times go - I appreciate that it probably contributes to site speed, but I don't think this is the root cause of our problems. We've always been very design/concept heavy and nothing has changed in this in the last few weeks - since we've been experiencing the intermittent problems anyway. I think as we improve our sites we'll look to move away from these problems - but thanks for all the advice it is most welcome.

The DNS issues are currently being investigated, so thanks for that - it may well be that we just need a couple of new servers and I believe this is in the offing. The tech guys are adamant this is an IIS issue so we'll see what comes of that too.

milonic: we are based in the UK yes - but we host the servers in house. It's odd though because entanet do host our 2 meg line - but this isn't widely used and shouldn't be affecting things in any way I can see. Thanks for the heads up though.

Thanks again guys and if you have any more advice I'd really appreciate it

Cheers

Matt

[coolguy27]

Slow site is results from many loading files like .css and javascripts limit delete some unnecessary javscripts and .css 'coz SEs love crawling the static and html, xml....

[Webnauts]

Did any one mention this?

<?php
ob_start ("ob_gzhandler");
?>