Google Wants Your Sitemap

[wrmineo]

WOW! is the first thing that comes to mind when reading the Google Blog this morning.

Google is actually asking for webmasters to submit sitemaps to help ensure Google has all of your current pages and, more importantly for webmasters, increase your coverage in Google.

They readily admit that this newest project will have one of two possible outcomes: fail miserably or succeed beyond our wildest dreams. However, this comes down to a judgement call for webmasters.

In my judgement, I'll follow Google's lead and hope for the best; they undertake nothing without calculating the cost-benefit analysis. Furthermore, Google loves Google - I don't see this hurting my site provided it's in order pursuant to their guidelines.

To make the prospect of the project even sweeter, they are offering a Sitemap Generator.

My suggestion would be to ask "how high" when the Big G says "jump" and do it now; no doubt, they'll be overwhelmed with submissions within hours.

Good Luck and Happy Ranking!

w®m

[flood6]

I just saw this a little while ago, too. There is also a slashdot story.

It's funny because most will agree that submitting your site to Google using the existing "submit url" method doesn't do much good. The idea being that if you get enough links to your site (page) Google will find it. If there aren't enough quality links to your site (page) Google doesn't care about indexing it. This seems to go against that philosophy.

Most of my sites don't have any problem getting crawled, but I've got a wiki that doesn't get indexed too well because there isn't much structure to how the pages are linked throughout the site.

My host has Python 2.1, and their xml-sitemap generation script needs 2.2. I guess I'll need to figure out another way to automate it.

[KeithO]

Tis interesting but we all consider it a prerequisite anyway.

[brian.mark]

We've been seeing fewer and fewer deep crawls, so I tried it out. I'll let you know what the results look like when I see a few spider counts.

BTW, that's a very spider-only sitemap that it makes. Seems funny that they're encouraging that. I thought we were supposed to make sitemaps for our visitors and it was just a coincidence that it also helped spiders. ;-)

BEGIN: BadIdea (next thing you know, they'll want us to cloak our sitemap for them) END: BadIdea

Brian.

[cbp]

Google do NOT want your site map!!!!

They want a particular type of XML feed from your site, that they have called Sitemap.

The only real benefit is to help Google crawl difficult to crawl sites.

Here is Google's FAQ's:
https://www.google.com/webmasters/si...cs/en/faq.html

CBP

[Weedy Lady]

I am trying to make the google site map, using the example they provide on their site. However, I know nothing (nada, zip) about xml, and am trying to do it by "filling" in the lines.

The problem is: On their example they list pages from a catalog, and I just have plain old .html pages. How do I list those........the instructions on the google page do not tell me how to list:

http://www.happydaycards.com/holiday...masredhat.html

They all have strange things like

http://www.yoursite.com/catalog?item...on_new_zealand

How does the above line relate to one html page?????

If someone can tell me the proper way to encode my individual pages I can start on all 800 of them. Just give me the proper code for the one page I've listed above, and I'll be off and running.

I am not going to try to use their generator because that is even more confusing to me, and there are quite a few pages on my site that I don't want included anyway.

Thanks.

[cbp]

I just noticed that Chris has posted an article about this:
http://www.webproworld.com/viewtopic.php?t=46621

CBP

[wrmineo]

Originally Posted by cbp

Google does NOT want your site map!!!!
They want a particular type of XML feed from your site, that they have called Sitemap ... Here is Google's FAQ's: https://www.google.com/webmasters/si...cs/en/faq.html
CBP

Thanks for the clarification! I only read what was on the Google Blog; should've looked deeper!

Thanks again!
w®m

[brian.mark]

Originally Posted by Weedy Lady

http://www.happydaycards.com/holidays_dec/xmasredhat.html

http://www.yoursite.com/catalog?item...on_new_zealand

How does the above line relate to one html page?????

They're exactly the same. Just substitute your static HTML page for the fancy dynamic URL and you're ready to go.

Brian.

[steve_avs]

I have tried various sitemap programs in the past without much success. Having a site with almost 40,000 pages makes things very difficult.

I'll be interested to hear from those with large sites as to how this works for them..

Regards,

Steve

[Conficio]

Hi all,
some interesting development indeed. Here are a few questions I do have though.

• How does this interact with robots.txt? Do they still obey the robots rules even if I list a page that is forbidden?
• What sense does it make to have an update frequency per page? See another post for a little discussion.
• How is the priority defined? Why would I define any low priority as a webmaster. I'd guess all pages are important to someone.

My feeling is this is a bit on the beta side as of yet.

K<o>

[Conficio]

I wonder why Google chose an "update frequency per page". I find that rather useless.

In the case of a static web-site I have a hard time to determine that beforehand. So I'll guess on the more frequent side (hourly or daily) and Google still waists its and my resources.

If I have a dynamic site, I will likely also create the sitemap dynamically and know even less on a per page basis how often I update this. Assume a catalog, how often do you think you update a particular item? There might be items you change multiple times a day and others not in months. If for example, you'd like Google to reflect a realistic "in-stock" counter, you want to have more like an hourly frequency. Again it likely waists Google's and my resources too much.

I'd rather have a frequency, how often they check my sitemap and find out the latest update stamp for each page. This would allow me to maintain those correctly and set a floor under their discovery time.

In reality it does not matter, how frequent a change is happening. From a webmaster's perspective it is of relevance how fast an update does get into Google. Has anybody said ASAP? Sure we all would like that.

However, if I tell Google to check on an hourly basis I don't want that for many pages. But I can tell googlebot a certain subset of pages in my catalog, namely the ones that I can reach through the daily, weekly, monthly, periodical. So if Google looks up my specific sitemap for this section on an hourly basis, I know that at the latest an hour after I updated that the periodical page Google takes notice and tell the world.

Another improvement would be to let Google know that I tend to update this page every Monday or on the third Tuesday of the month. Think of regular meetings and the protocols you are posting afterwards, as well as the invitation to the next meeting.

Just my initial take. Any opinions, comments?

K<o>

[brian.mark]

Since I submitted, Google hasn't spidered their normal 100 pages/day. Instead, it's been 1. I hope this wasn't a trap for over-excited webmasters.

Brian.

[brian.mark]

I was scared when I saw numbers lower than normal, then the storm hit. 4,000+ pages in a day. Looks like it really did get more of our pages indexed.

Brian.

[wrmineo]

I hope it works out; I was scared that I'd pointed you into dangerous waters after I saw your earlier post.

[ctabuk]

In point of fact, Google now has students employed from India to the Netherlands who's job it is to review sites. They are creating a human read Directory of sites with valid content. Mine has leapt up on Google after I had discussions with the UK Government about certain sites within the Financial Services sector who had illegal sites. Many of these have been manually removed by Google.

[MathsIsFun]

I read recently about that "Google has humans reviewing websites" topic, and a reply from (I think) GoogleGuy said that it was part of a quality evaluation exercise and nothing to do with the SERPS of actual websites.

Sorry I can't find harder data, just relying on memory that it was a bit of a red herring.

[wrmineo]

Originally Posted by MathsIsFun

... part of a quality evaluation exercise and nothing to do with the SERPS of actual websites.

Correct, G is adamant that humans do not, cannot, and will not be be able to "hand" move a site's ranking.

However, I do believe that with the human eval and QA they're experimenting with, coinciding with the typical removal of spam and unethical sites, that ctabuk's point of getting a boost in SERP is very valid.

With less junk sites stealing positions, it makes sense that the remaining, valid sites will naturally move up in the pecking order. It a matter of Math isn't? :) (Sorry, I just couldn't resist that!)

w®m

[MathsIsFun]

LOL! Yes, indeed, it is mathematically sound!

Point taken. If we take the time to report spam, and Google are now taking it seriously, then we not only can improve our own positions, but we make the search better for users (and possibly even save them from sharks).

I just hope the reviewers really do look at sites, otherwise spammers could turn the tables and report good sites (from multiple accounts to make it look like overwheling evidence) to try to improve THEIR ranking.