Google Sitemaps: RSS For The Entire Website?

Chris · #1 (**permalink**) 06-03-2005, 12:10 PM

Well, no, not really. But Google Sitemaps does employ XML technology in order to provide its program members the opportunity to have their site crawled after they make updates or alterations.

Before we get ahead of ourselves and in case you haven’t heard, yesterday, Google launched Sitemaps, a “collaborative crawling” service designed to keep Google informed of modifications to your web site so their search index can reflect these changes… or as Rusty call it, a free pay-for-inclusion program.

Sitemaps works by taking advantage of XML and RSS capabilities. By placing XML code on the web server, you inform Google of when changes occur, and they respond by crawling the updated pages and making the necessary updates to the search index. Over at the Google Blog, Engineering Director Shiva Shivakumar indicated why Google launched Sitemaps:

“Initially, we plan to use the URL information webmasters supply to further improve the coverage and freshness of our index. Over time that will lead to our doing an even better job of delivering more search results from more websites.”

Shiva also gave an extensive interview to Danny Sullivan over at the SearchEngineWatch Blog. In it, Shiva iterates that the Sitemaps program’s current beta state; he won’t guarantee each submitted URL would be crawled. He did indicate that this was something they were working toward, however.

As mentioned, in order to participate in Sitemaps, you have to have a Google account and you have to place an XML file on the webserver being used by your site. This is done in order to inform Google’s crawlers of what URLs to look for and how often these pages change. As pointed out by Rusty, over at SocialPatterns.com, SEM Michael Nguyen broke down an example of his Sitemaps’ XML code, line-by-line; in order to shed some light on what’s actually being done.

The XML file must also the URL of each page you want to be in the Sitemaps program. If you have four pages that undergo frequent change, all 4 page URLs should be listed, if you have an entire site that you want included, you have to include the URL of each page. By employing the changefreq and priority XML tags, you can also indicate how important each page is and how frequently the page changes.

After the XML is complete, you must submit it to the Sitemaps program. This is where the Google account comes in. Once the URL of the sitemap is submitted, your task is complete.

There are a couple of methods you can use in order to get an XML sitemap. A sitemap generator can be downloaded from Google or you can develop one. The generator is an open-source Python file that has to be uploaded to the webserver. According to their FAQ the sitemap generator “can create sitemaps from URL lists, webserver directories, or from access logs.”

You can also develop your own XML sitemap if you so choose. This will have to be submitted as well. The final method Google accepts is a text file containing the URLs you want in the program. Obviously, this method is saved for those who have little-to-no experience dealing with webservers or structural web alterations. It also seems like the text files will be given the lowest priority, at least until the program is off and running.

As to whether or not you should be taking part in the Google Sitemaps program is quite simple: if search engines play any role in your business whatsoever, you should be apart of the program. Having Google’s (or any other search engine for that matter) index reflect changes in your site quickly will only benefit your search engine presence. Or as Nathan Weinberg says, it’d be stupid not to.

An additional area of interest is that Google made Sitemaps as open-source as possible… at least on the XML end. By making the sitemap generator in Python and releasing it under the Attribution/Share Alike Creative Commons license, Google is only furthering their embrace of open-source. This also allows the program to be adapted in order to support other search engines.

Update: Someone who emailed me installed the sitemap generator on his webserver, and evidently the server went boom... or it at least was overtaxed. Here's a quote from his post discussing the event: "Running it brought down my 3200MHz Pentium 4 running Debian Linux and 2 GB of RAM."

Read Theo's report and see for yourselves.

MeanSEO · #2 (**permalink**) 06-03-2005, 01:59 PM

Obviously anyone not taking advantage of this new feature aught to have their head examined!

strum4life · #3 (**permalink**) 06-03-2005, 02:25 PM

This is great, but it will take me forever to accomplish for all my sites.

strum4life · #4 (**permalink**) 06-03-2005, 03:00 PM

Quote:

Originally Posted by CRich

Here's a quote from his post discussing the event: "Running it brought down my 3200MHz Pentium 4 running Debian Linux and 2 GB of RAM."

If this is the case, screw the SiteMap Generator. If you're using PHP and MySQL you can easily create a dynamic XML file that will never bring down a Pentium 4. Then create a cron job to tell Google when your sitemap has been updated. My cron job looks like this:

Code:

* 0 * * * curl http://www.google.com/webmasters/sit...om/sitemap.xml > /dev/null

This tells Google that your sitemap has been updated every hour.

espectations · #5 (**permalink**) 06-06-2005, 02:44 AM

Hi, I clicked on the links in the article and none of them worked?

Did Google take down the pages since launch of this article?

There are a lot of opportunities with this sort of thing - I am glad they are moving in this direction ......

KeithO · #6 (**permalink**) 06-06-2005, 11:12 AM

only problem is we don't have python installed on our servers. :(

ReviewGolf.com · #7 (**permalink**) 06-06-2005, 11:41 AM

Actually It's not a such a bad idea, especially if some of the content you'd like to be spidered is dynamically driven...

Spectur · #8 (**permalink**) 06-08-2005, 09:01 AM

I tried it but once again python is not installed on my isp.. Its a great Idea but why not give a dtd file and lets us create and submit a sitemap xml to google ?

This would be an interesting option.

just a hint to the goggle guy not everyone has access to the command line on there webserver..

flood6 · #9 (**permalink**) 06-08-2005, 12:22 PM

Quote:

Originally Posted by Spectur

I tried it but once again python is not installed on my isp.. Its a great Idea but why not give a dtd file and lets us create and submit a sitemap xml to google ?

You can generate and submit an xml sitemap using whatever means you want. They supplied the python code to get the ball rolling.

The specs for the format they want are here.

greeneagle · #10 (**permalink**) 06-10-2005, 10:21 AM

This protocol seems to have the distinct possibility of tilting in the favor of an elite group of professional spammers.

Ken

se-survivor · #11 (**permalink**) 06-13-2005, 01:53 PM

Quote:

As mentioned, in order to participate in Sitemaps, you have to have a Google account ...

I'm not so sure about this. You may be able to generate a simple .xml file with a site like this
http://www.googlesitemap.info/thesimplerway/ (ignore steps 1 & 5)
or
http://www.better-business-directory...-generator.htm

and then ping the server anonymously without a Google account:
https://www.google.com/webmasters/si...en/faq.html#s4

..for those of us worried about being assimilated by the Borg's database...which is what this is really all about.

Chris · #12 (**permalink**) 06-14-2005, 05:31 PM

se-survivor:

you are correct sir. my mistake.

Google says:

6. Do I need to sign up for a Google Account?

You don't need an account to generate and submit a Sitemap. However, we encourage you to sign up for an account so that you can track the status of your Sitemaps and view diagnostic information for your submissions. Having an account will not affect your site's ranking within our results. If you already use Gmail, Groups, My Search History, Alerts, or Froogle Shopping List, you already have a Google Account and can sign in with your existing account to use Google Sitemaps.

kgun · #13 (**permalink**) 06-15-2005, 10:31 AM

Is RSS (readers) the next source of fraud / black hat.

There is one advantage with browsers, they are mature. Netscape invented the HTTPS protocol as far as I know.

There are now a lot of various RSS and atom readers and aggregators. Will they be used for creative inventions?

If you use a reader or aggregator, check the company. Call them and talk to them. Myself I use NewsGator. Talked to them for an hour.

What can be exported / imported via RSS (XML).

KBleivik
http://multifinanceit.com/
http://www.multifinansit.no/

P.S. Have you seen the sites that change content of other sites? It is not so easy on secure sites.
Possible to manipulate news (superimpose your own news)? I only ask.

se-survivor · #14 (**permalink**) 07-12-2005, 12:10 PM

Hey Crich:

Guess what. They closed the door. Now you're right and I'm wrong. World domination is back in play...

6. Do I need to sign up for a Google Account?

You need an Account to submit a Sitemap the first time. This allows you to track the status of your Sitemaps and view diagnostic information for your submissions. If you already use Gmail, Groups, My Search History, Alerts, or Froogle Shopping List, you already have a Google Account and can sign in with your existing Account to use Google Sitemaps. Otherwise, you can sign up for one.

They totally removed the anonymous option.

BTW, I noticed that MSN search indexed my page changes almost the same day I put up my sitemap, where Google is still ignoring it. I think MSN search is loving this.

Chris · #15 (**permalink**) 07-12-2005, 02:37 PM

se-survivor:

nice of them to switch up like that. :-D

TheWebDoctor(tm) · #16 (**permalink**) 07-16-2005, 07:10 PM

After a month of using Google Sitemaps, it's provided no real value. In fact, doing an A-B test with two sites launched the same day, the site without the Google Sitemap has found more of its pages in the Google index than the other site.

We're actually expecting the Google Sitemaps to be no more advantageous than the Add URL for Google.

brian.mark · #17 (**permalink**) 07-16-2005, 10:51 PM

Quote:

Originally Posted by TheWebDoctor(tm)

After a month of using Google Sitemaps, it's provided no real value. In fact, doing an A-B test with two sites launched the same day, the site without the Google Sitemap has found more of its pages in the Google index than the other site.

We're actually expecting the Google Sitemaps to be no more advantageous than the Add URL for Google.

We have a site that has been around since 1998. We've been having trouble getting deep crawls (even with static HTML pages), but it's all indexed now that we gave them a sitemap.xml file. For us, it's been huge.

Brian.

Spectur · #18 (**permalink**) 07-19-2005, 11:10 AM

So I was digging around in the Dreamweaver exchange and found a cool addon to allows you to generate a sitemap.xml for google no more need for python :)

so if you have DW mx 2004 you can now generate sitemaps very easily...

Chris · #19 (**permalink**) 07-19-2005, 11:17 AM

Quote:

Originally Posted by Spectur

So I was digging around in the Dreamweaver exchange and found a cool addon to allows you to generate a sitemap.xml for google no more need for python :)

so if you have DW mx 2004 you can now generate sitemaps very easily...

gotta link?

Spectur · #20 (**permalink**) 07-20-2005, 12:33 PM

You will need to test up an account to dl it . But accounts are free.

http://www.macromedia.com/cfusion/ex...ver%20Exchange

incusa1 · #21 (**permalink**) 07-21-2005, 09:24 AM

Can anyone help? We are trying to findout how to get
"featured" on Webproworld for correction, up-to-date coding, and how to incorporate webproworld.com in our own website for page results. We really need some critical input. We rank well on MSN and Google, however....not good enough. Incorporate USA or the website at http://www.inc123.com original that ranks well is http://www.incorporate-usa.com since 1994 any comments....please help

jm

brian.mark · #22 (**permalink**) 07-21-2005, 12:50 PM

Quote:

Originally Posted by incusa1

Can anyone help? We are trying to findout how to get
"featured" on Webproworld for correction, up-to-date coding, and how to incorporate webproworld.com in our own website for page results. We really need some critical input. We rank well on MSN and Google, however....not good enough. Incorporate USA or the website at http://www.inc123.com original that ranks well is http://www.incorporate-usa.com since 1994 any comments....please help

jm

Submit your site for review. I think they just randomly pull one from that category that was recently submitted as the "Featured" site.

Brian.

dealsaroundus.com · #23 (**permalink**) 08-04-2005, 01:07 AM

Interestingly mixed comments! I want to try my self to see the results. Is it going to consider Page Rank for timely crawling?

Certainly I will post my observation and comments later.

Spectur · #24 (**permalink**) 08-04-2005, 08:56 AM

well here is an insteresting thing.. Before I did my sitemap.xml for google I had a PR of 0 now that I did the sitemap I jumped to a PR of 2 over night.... interesting result...

stoni · #25 (**permalink**) 08-16-2005, 01:29 PM

Just added <link rel="sitemap" type="application/xml" title="Sitemap" href="http://ramonantonio.net/sitemap.xml" /> to the head tag.
It would be nice if all search engines make use of this technology.

Alli · #26 (**permalink**) 08-21-2005, 03:05 PM

Thanks for the great info. Here are some helpful tools that I've found around the internet pertaining to Google Sitemaps.

https://www.google.com/webmasters/si...generator.html - Google SiteMap Generator
http://www.auditmypc.com/free-sitemap-generator.asp - Free SiteMap Generator

http://www.rjsoftwares.com/google_sitemap/ - Google Sitemap Script - This script automatically updates and is based on PHP.

brian.mark · #27 (**permalink**) 08-21-2005, 04:24 PM

Quote:

Originally Posted by stoni

Just added <link rel="sitemap" type="application/xml" title="Sitemap" href="http://ramonantonio.net/sitemap.xml" /> to the head tag.
It would be nice if all search engines make use of this technology.

There are reports of other bots hitting those files, so I don't think it'll be long.

Brian.

boohlick · #28 (**permalink**) 08-22-2005, 09:11 AM

Quote:

Originally Posted by TheWebDoctor(tm)

After a month of using Google Sitemaps, it's provided no real value. In fact, doing an A-B test with two sites launched the same day, the site without the Google Sitemap has found more of its pages in the Google index than the other site.

We're actually expecting the Google Sitemaps to be no more advantageous than the Add URL for Google.

Google sitemap is for easy crawling of all of your page, it doesnt gives you any gurrantee to be index or to have good SERPs.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode