Googlebot and MSNbot not visiting

jonathanbriggs · #1 (**permalink**) 09-13-2005, 09:08 AM

Very puzzled. Done lots of this before and not had these problems.

We launched http://www.foodparcels.co.uk and after one quick visit in April - both Googlebot and MSNbot have disappeared. They are visiting all our other sites including our company http://www.othermedia.com which points at Food Parcels.

We have also created inbound links from a few directories and other sites but these stubborn robots refuse to even sniff the home page. It has been over 3 months now and the client is getting jumpy.

Any suggestions? Have we done something wrong? Have other people had similar experiences?

J.

onelife · #2 (**permalink**) 09-13-2005, 10:30 AM

Hi there,

you seem to have a lot of links from this blog http://dev11.otherworks.com/theother.../06/YYakKAgwik - dubious how relevant they are.

As you will see from the url above it seems filled with garbage and adsense ads, most of the pages don't offer any topical text - just more garbage.

Certainly not what you would need linking to your site, which offers classy food parcels.

I think you might want to get some decent links with anchor text pointing to the site, maybe you have already but they aren't showing in Google yet.

Just my opinion

jonathanbriggs · #3 (**permalink**) 09-13-2005, 10:35 AM

Thanks onelife - appreciate the comments.

I think the guy who runs the blog you have mentioned might think 'garbage' is a little strong :-) but you are right that by adding our site to his blogroll he has perhaps burdoned us with lower quality links than we need. Do you think Google/MSN would penalise us for that?

onelife · #4 (**permalink**) 09-13-2005, 12:00 PM

Hi,

Maybe garbage is a little strong but there is only some many times you can read the word 'penis' before you think the worst.

I firmly believe that both MSN and Google are constantly looking at ways to rank and justify search results. Part of this I'm sure is with high quality links from established or reputable sites.

Unfortunately blogs with hundreds of repeated 'penises' and mobile phone model numbers are not going create this. I would think Google and MSN may view these links as being from a bad neighbourhood, as there really is a lot of them.

I would try and get them removed from the blog. Get more new 'on topic' links from reputable sites with your keywords in the anchor text and submit your sitemap to google.

Hope this helps

jonathanbriggs · #5 (**permalink**) 09-13-2005, 12:18 PM

I did not take the garbage comment badly at all and on looking at Tom's blog it seems to have been the victim of severe blog spam. I have emailed the blogger and will get him to remove it.

Going back to my orginal posting. My concern was that the things that normally work (good quality links) appear to have been much less successful than normal. Usually I add a couple of links to our corporate site, to a few well chosen directories and to a couple of our other relevant clients and bingo - in comes the spider and the pages get into the index.

Then I find more links, add more content, more links, more content etc watching the site gradually climb the ratings for appropriate keywords. It can certainly take a few months but in this case I can't make first base and get the bots to start feeding.

Puzzled.

Orion · #6 (**permalink**) 09-13-2005, 06:50 PM

I quickly checked your doctype declaration and ran the URL on the W3C validator. There are 17 errors. I'm not saying they have anything to do with what the problem is, however it definitely can't hurt to have the pages all 100%.

The site is listed in google but it's not cached. http://www.google.com/search?q=foodp...en-US:official
and it's listed in MSN too.

Are your other sites .co.uk? and are you checking your rankings in the localized versions of each engine?

Another large item may be that there is very very little content on the page itself. Try adding a couple hundred word paragraph that's rich in keywords, makes sense and can aid towards the marketing goals of the page/site.

Beyond that setting quality links is a good endevour to rank you well and bubble the site up through the ranks.

hope something here might help...

Weedy Lady · #7 (**permalink**) 09-13-2005, 07:21 PM

This is pretty elementary, but since you didn't mention it, have you tried Google's sitemap service? Although submitting a sitemap is in no way guaranteed to get you listed or ranked, it does let Google know that the site is there. It doesn't necessary flag any other SE.

I do mine manually and resubmit it when necessary, so I always get feedback of when they last downloaded the site map. If you use their automated system you will not get this feedback.

**incrediblehelp** · #8 (**permalink**) 09-13-2005, 08:44 PM

http://dev11.otherworks.com/theotherblog/

Does this person use autoblogging software?

It also concerns me that the only IBL's that Google really shows are from a couple of blogs:

http://www.google.com/search?hl=en&q...=Google+Search

MtraX · #9 (**permalink**) 09-14-2005, 03:13 AM

Hi

I also agree that the quality if your inbound links are important. Have a look at the directory thread. There are many directories where you can list your site for free. Don't overdo it though, or get a massive amount of inbound links at one time as this will be seen as further spam. Also take note of the anchor text used on those links and include your main keywords, making sure your page titles also reflect those inbound links' anchor text.

Regards
MtraX

clivemcg · #10 (**permalink**) 09-14-2005, 03:18 AM

I have recently used the Google sitemaps for some clients sites and this resource is handy for automatically creating the .xml file for you.

http://www.blocklayer.com/googlesitemap/

Don't forget to Verify the map by getting the Google code and turning it into an empty .html file and put it into the root folder where the .xml file sits. It's easy to think you've completed all the steps once you've uploaded the .xml file

Will it help? I can't say yet as it's early days for me but would be interested to hear if it does.

mrmdobrien · #11 (**permalink**) 09-14-2005, 03:23 AM

There is a severe lack of content on this sites home page.

Also one of the most important tools you can optimise on your site is the title tag, and you have totally under used it ('Food Parcels: Home page').

You have gone a bit crazy with div's e.g

'<span class="clear"></span>
<div class="col765">
<div class="middle">

<div class="col560">
<div class="col560">
<div class="strapline">'

It looks obvious that your coding needs a good clean up, just look at all the unneeded code above as an example.

mktplace · #12 (**permalink**) 09-14-2005, 08:38 AM

You don't appear to have a robots.txt file.

If you do, it is throwing errors, as your error page comes up when trying to access it.

DrTandem1 · #13 (**permalink**) 09-14-2005, 09:12 AM

I'm not sure why the 'bots haven't been back to visit your site. However, it has nothing to do with blog spam as long as the blog isn't hosted by your site/server. It also has nothing to do with your page's coding. I'm all for validation, but I can show you sites with horrific coding that are number one for a generic term on Google.

The suggestions of a site map and improving your title tag are good, but they are not a reason for the lack of spider visits. Neither is not having a robot meta tag. If it's missing, they just visit and do what they are programmed to do.

I suspect the trouble may be that you have two domains off the server pointing to the UK site. Your other one, foodparcels.com is also on:

DNS21.REGISTER.COM 216.21.234.81
DNS22.REGISTER.COM 216.21.226.81

as is the UK site. When going to the dot com URL, it redirects (how?) to the UK site. This is most likely perceived as a doorway page. Big time penalties are possible for that.

I'm not sure why your client has elected to go with the UK domain rather than the dot com as both were created at the same time. Maybe it's legal issues. Anyway, if you want to use the UK URL, remove the dot com version from the same server and simply have it forwarded. Then wait some more. Patience.

If you have any other URLs that are doing this, also remove them from that server. If they are not on the same server, be sure you are using a 301 redirect.

Even though your redirect may be simply an honest attempt at preventing copy-cats, I think the search engines view it as a doorway page.

etechsupport · #14 (**permalink**) 09-14-2005, 09:49 AM

I think if you do a considerable amount of changes to your homepage everyday you'll get Google to check your site every day and new pages will be added, if you add new content Google will figure it out and deep crawl more often.

adacprogramming · #15 (**permalink**) 09-14-2005, 10:37 AM

Google site maps are good, however they only help you with Google and you have to make sure you keep them up to date.

I prefer a text link in my footer going to my own sitemap page where I have text links to each page I want the bots to parse. This works for all spiders and at least for me is easier to remember to update.

Getting your site listed in a number of directories (even forums like this) will keep reminding the bots to hit your pages. Bear in mind MtraX's warning. Only do a few at a time.

DrTandem1 · #16 (**permalink**) 09-14-2005, 10:54 AM

Quote:

Originally Posted by etechsupport

I think if you do a considerable amount of changes to your homepage everyday you'll get Google to check your site every day and new pages will be added, if you add new content Google will figure it out and deep crawl more often.

If the 'bots aren't visiting why would you think that changing anything on the Home page would accomplish the mission of having them visit? In other words, they are not visiting for a reason. Simply changing content and coding (SEO) will do nothing to prompt a visit. The search engines don't feel a change in "the force" and decide to visit. They have detected something that they apparently find objectionable.

Jonathan has asked a good question. I can show you plenty of sites that have very little content and are high in Google's SERPs. I can also show you sites with a great number of links pointing to them that are not found high in the SERPs.

I urge Jonathan to explore the multi-URL/same server issue. I have seen sites that had the same problem and once the doorway page or improperly redirected URLs were removed, the site made a huge gain in the SERPs after their next visit.

While I agree that the page is not as search-engine friendly for whatever (no focus) his keyword targets are, they are not a reason for the lack of crawling his site. Yes, I would refine the title tag, and add descriptive alt attributes to the images, but first I would eliminate potential penalties.

DrTandem1 · #17 (**permalink**) 09-14-2005, 10:59 AM

Quote:

Originally Posted by adacprogramming

Getting your site listed in a number of directories (even forums like this) will keep reminding the bots to hit your pages. Bear in mind MtraX's warning. Only do a few at a time.

Yes, and if every time the 'bots visit they see a doorway page and/or an improper redirect, it will confirm to them that they do not want to crawl the site. I don't care how many links point to a doorway page, it will still be considered a doorway page.

Dan Bigs · #18 (**permalink**) 09-14-2005, 11:12 AM

I took a look at your back links and they are not relative to your site. How long has the site been up and running and how long has the domain name been owned for?

DrTandem1 · #19 (**permalink**) 09-14-2005, 11:37 AM

Quote:

Originally Posted by Dan Bigs

I took a look at your back links and they are not relative to your site. How long has the site been up and running and how long has the domain name been owned for?

His domain has been around since January. Both of them on the same server. I don't believe the issue is SEO. Again, he has a doorway page and/or improperly redirected site. Google takes an especially dim view of this according to their own webmaster information pages.

jonathanbriggs · #20 (**permalink**) 11-12-2005, 11:54 AM

Well, we found the answer! And it was a mixture of changes in the way we serve and the way Google and MSNbot index our sites.

It used not to matter whether you included a "robots.txt" file for your site or not. Robots.txt is normally used to tell a search engine spider that you do not want parts of your site crawled. The non-existence of this file implied that you did not mind the crawler looking everywhere. So far so good.

Well behaved spiders always check to see whether the robots.txt exists. In food parcels case it did not but this should not have mattered.

But, the way we serve our sites has changed and instead of simply reporting that the file did not exist our server reported an error. Google noticed the error and decided not to look any further. Solution - add a robots.txt file!
It has taken us weeks to find this error and it was so simple and staring us in the face. The lesson is simple; don't assume anything about the way your technology works particularly when it comes in contact with someone else's technology.

Well done to the person who spotted that! And its good to be in the index at last.

Jonathan