Google taking very long to index page

[cppgenius]

Google is very lazy indexing pages of our site (taking more than a month now). It has a sitemap that is valid, it has an html version of the sitemap, there is nothing wrong with the internal linking structure and Google has downloaded our sitemap and crawled our index page on several occasions now, but fail to index new pages and update existing ones that have been updated with new content.

On each page there is a dynamic banner on the left that displays one of our top 3 performing products each time a page is loaded or refreshed. I have done this with PHP and not Javascript, so search engines may get different code in this part of the page each time they crawl our site. It's only a little piece of code, an <a href> statement containing a different url each time and an <img> statement containing a reference to a different image file each time.

Could this perhaps make the page too 'dynamic' to Google's liking and that's why they are taking so long to index/update the page?

[jtracking]

ya i always wonder as well if you have a rotating banner or block of text that changes with each page load or something would that effect your standing in the serps...

[incrediblehelp]

Originally Posted by jtracking

ya i always wonder as well if you have a rotating banner or block of text that changes with each page load or something would that effect your standing in the serps...

I doubt this is an issue. Google crawling schedule is somewhat dependent on your number and quality of IBL's. How does that look for you?

[cppgenius]

incrediblehelp, the IBL's for the specific site in question are not so good at the moment.

I downloaded gsitecrawler today to see how it crawls the site and it reported the last modified date of each page as today's date. That made me wonder, Google probably receives the same dates, so there is no quick way of telling which pages got updated recently, so do you think it will help by adding the dc.date.x-metadatalastmodified tag to each page and have the sitemap updated with these dates?

[Webnauts]

If you would tell which URL you are talking about, I could tell more. Even if I wouldn't, adding the link here will be soon followed by Google and your site would be visited.

[Steven1976a]

We are also experiencing the same with our site Rattan Home. A page from around 6 weeks is showing in the cached.

Any comments as to why this is? Is there anything in the code blocking it from being indexed?

Rattan-Home.com

[Orion]

IBLs, Google sitemap and openDMOZ seem the best ways to get into google. Make sure you do all of them.

I find submitting to vertical directories seems to help too.. not all end up qualifying as quality IBLs but they do seem to help overall.

[cppgenius]

Webnauts the site is http://www.liafin.co.za. When I started working on this site in 2005, it did pretty OK, not exceptionally well, but absolutely better than now. The problem was that the whole approach of the site was wrong and did not pose any long term possibilities for success so two months ago I completely revamped the site, changing the main purpose and goals of the site (I placed a lot of emphasis on what the site can offer to the visitor, ie useful, interesting content).

The layout and navigation stayed pretty much the same, but the pages have been optimized and improved for better search engine placement and better keyword targeting, but the improved pages seem to do worse than the previous unoptimized pages at the moment. It is really strange (ironic actually), because I have redesigned the site with the visitor as main priority, but without neglecting the important SEO elements, but still the rankings and visits (and the indexing of new and updated pages) have taken a sharp decline.

I have added the meta tags and generated a sitemap containing these last modified dates of each page. It will be an interesting experiment to see how fast Google react to these changes. On the other hand if they update our index in the next few days/weeks, it might just be that Google suddenly decided to update their index at the same time when we uploaded the new sitemap, you'll never now, would you?

Thanks, I appreciate the help and advice.

[Webnauts]

cppgevius, here are some things I can tell on the fly:

1. Your XML sitemap is not valid. Google Sitemap Validator

2. I would write the robots.txt all the way around as it might be confusing the bots. I mean like this:

Sitemap: http://www.liafin.co.za/sitemap.xml
User-agent: *
Disallow: /link.php

3. Add some more content on your index page.

There are for sure some more issues on the site, but I guess others can help you out here.

Good luck.

[TrafficProducer]

taking more than a month now

I believe Google take at least 2 months to index sites, if at all.

[Veikoh]

I have seen some complaints about Google indexing before, so decided to answer as I do not have any such kind of problems. After I publish new blog entry it will be shown up after 2-3 days so maybe the problem is somewhere else.
1. Did you refresh the content or is the content same? How frequently you are refreshing your content?
2. Do you have same incoming links? Maybe publishing it with tags to content aggregators will help? Also allow user to bookmark your site with del.icio, technorati, etc. All those links are considered as incoming links for search engines.
3. What kind of key phrases you expect your site to come up? Maybe changing those will help. One of google algorithms is to follow users behaviour. So, if you didn't get much clicks Google thinks, it's not relevant content for particular search.
4. I took a look to your sitemap. There is no XML code at all for sitemap? The proper format is:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex lns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>http://www.example.com/sitemap1.xml.gz</loc>
<lastmod>2004-10-01T18:23:17+00:00</lastmod> </sitemap> <sitemap>
<loc>http://www.example.com/sitemap2.xml.gz</loc>
<lastmod>2005-01-01</lastmod>
</sitemap>
</sitemapindex>

Hopefuly it helps. Veiko

[Webnauts]

Originally Posted by TrafficProducer

I believe Google take at least 2 months to index sites, if at all.

My sites never took longer than 4-7 days.

[ctabuk]

I am only a humble marketer - yet with just one link I got a 12 hour site virtually top of page 2 on Google UK yesterday.
Open Market HomeBuy scheme information for Key Workers

It's all about titles. Now we start adding pages and a slow links campaign.
Keyword Open Market Homebuy - What else!!!

[Webnauts]

Originally Posted by ctabuk

I am only a humble marketer - yet with just one link I got a 12 hour site virtually top of page 2 on Google UK yesterday.
Open Market HomeBuy scheme information for Key Workers

It's all about titles. Now we start adding pages and a slow links campaign.
Keyword Open Market Homebuy - What else!!!

If the domain or site/site theme is new, don't expect to be long on the second page though.

[ctabuk]

Sorry to have to disagree John - but I did the identical thing with Shared Ownership Mortgages

It too reached page two on day one and is now on page one where it has been for 2 months. It took ages for MSN to index and when it did it went top on live.com and page one on Yahoo.

Initially, you are correct that it 'might' drop -but that is where sensible timed linking and adding content comes into its very own.

URL titles on short one page sites really does work.

[Webnauts]

Originally Posted by ctabuk

I am only a humble marketer - yet with just one link I got a 12 hour site virtually top of page 2 on Google UK yesterday.
Open Market HomeBuy scheme information for Key Workers

It's all about titles. Now we start adding pages and a slow links campaign.
Keyword Open Market Homebuy - What else!!!

OK. Lets look at this closer:

Keyword Discovery results:
Open Market Homebuy Activity (Estimated): 38 Web-Wide search engine queries per day! Should I shout out loud WOW?

---

Overture (UK) results
Open Market Homebuy - No searches at all!

---

Wordtracker (UK) results
Open Market Homebuy - No searches at all!

---

Google UK Competition 36,300 pages!

Can you please explain where do you see the great success?

Thanks.

[cppgenius]

Webnauts I applied all the changes you suggested. Thanks for pointing that out to me, I never thought of checking the sitemap for validity, coz I thought gsitecrawler would do that for me, well I guess this just proves that it doesn't. It's strange though, Google already downloaded the invalid sitemap and reported its status as OK.

Why doesn't the sitemap validate with the 0.9 schema, even though Google recommends this schema for valid sitemaps? https://www.google.com/webmasters/to...emapValidation

The order you suggested for robots.txt makes better sense, I agree and it can't do any harm changing it anyway.

Yes, adding content to the main page will help and there I'm guilty as charged, didn't have much time to add new content to the site lately, so I know that is one of the reasons why the rankings are going so slow, but I first want to get past this indexing hurdle to see if the changes I made to the current pages were worth the wile. Without the updated content in Google's index, I can't do much to see if the changes worked. I'm only afraid of doing a lot of work adding new content only to discover that some of the new techniques I'm using aren't working that well and I have to do everything over again. At the moment it is a process of trial and error until I get it right and then implement these methods with all the new content added to the site.

Oh and Veikoh thanks for pointing out that there is no entry for the sitemap itself, so I added that as well. (I think I forgot to add the .xml file type to gsitecrawler's settings, so that's probably why it missed my sitemap link).

Wow, its amazing to see how easy it is to look over the simplest of things. Actually makes me feel pretty stupid to see that I'm not even doing the basics right. LOL!

Now lets wait and see what happen. Thanks for the help.

[Webnauts]

Another tip. Add you site to this directory. That got my site seoworkers.com indexed within 4 days: Webotopia Business Directory: Business Web Directory

[ctabuk]

Originally Posted by Webnauts

OK. Lets look at this closer:

Keyword Discovery results:
Open Market Homebuy Activity (Estimated): 38 Web-Wide search engine queries per day! Should I shout out loud WOW?

---

Overture (UK) results
Open Market Homebuy - No searches at all!

---

Wordtracker (UK) results
Open Market Homebuy - No searches at all!

---

Google UK Competition 36,300 pages!

Can you please explain where do you see the great success?

Thanks.

Well I could start by saying that that was a very rude post.

Open Market Homebuy is a brand new Government initiative - any decent website marketers job is to look at markets as they evolve, create the sites ready for searches before they start to happen.

Like I said - I am only a humble marketer.

[ctabuk]

So, Google