New Posts = Instant Crawl by Google

uberanimal · #1 (**permalink**) 01-29-2008, 01:32 PM

A buddy of mine keeps asking me why is it when he does a new post on his wordpress blog site, which is about a year old now, google will crawl the new post instantly without any hesitation. I was trying to do some research with this because it only happens to certain sites. Why is this? What have you guys found out about this happening? How does google pick which sites do index the post instantly on?

carbonize · #2 (**permalink**) 01-29-2008, 02:18 PM

Possibly his WP generates a sitemap and notifies Google instantly of updates. I know my Serendipity does.

**crankydave** · #3 (**permalink**) 01-29-2008, 02:39 PM

Google picks up the feed. If you do a "blog" search you'll see that the blog posts get picked up immediately. They like to add this to their regular results rather quickly ("minty fresh") and then go back and actually crawl and index the page later.

Dave

uberanimal · #4 (**permalink**) 01-29-2008, 03:21 PM

Quote:

Originally Posted by crankydave

Google picks up the feed. If you do a "blog" search you'll see that the blog posts get picked up immediately. They like to add this to their regular results rather quickly ("minty fresh") and then go back and actually crawl and index the page later.

Dave

So google is pulling the feed first now? hmmm.. this is pretty new isn't it?

chowell · #5 (**permalink**) 01-29-2008, 04:36 PM

The WordPress blog has a "pinging" feature built-in that notifies a pinging engine (which notifies Google) immediately when a new post is made.

Your friend's site is merely benefiting from the insight of the WordPress creators SEO knowledge.

There are other plugins you could add to a WordPress blog to automatically create XML sitemaps & to notify Google of changes to them, but it doesn't come installed as a default, so your friend probably isn't using this method.

bj · #6 (**permalink**) 01-29-2008, 04:37 PM

If the Wordpress installation is set up to ping the update services, this creates instant backlinks, and near instant spider activity.

admin/options/writing and scroll to the bottom.

**wige** · #7 (**permalink**) 01-29-2008, 04:47 PM

Not sure how new it is. I added feeds to my non-blog website at customer request several months ago, and not only does new content get indexed very quickly, but we also perform quite well on Google's blog search. Also, if the blog is set to ping Google when it is updated, that may trigger the addition of the new content almost immediately as well.

Orion · #8 (**permalink**) 01-29-2008, 05:04 PM

Quote:

Originally Posted by bj

If the Wordpress installation is set up to ping the update services, this creates instant backlinks, and near instant spider activity.

admin/options/writing and scroll to the bottom.

I've had good success with this too.. I have a related question.. does anyone have or know of a fairly comprehensive list of update services that are recommended to subscribe to? I found it quite a time consumer the first couple times searching them all out.. Some are obvious, but I'm sure there's a ton I'm missing.

bj · #9 (**permalink**) 01-29-2008, 05:18 PM

The link to the Wordpress docs that is right in that section in the Wordpress admin interface leads to a list that is kept pretty up to date.

Orion · #10 (**permalink**) 01-29-2008, 06:06 PM

DOH! .. figures got a question check the help files.. ;o) Thanks!

datetopia · #11 (**permalink**) 01-29-2008, 07:20 PM

We have a google news alert for the keyword "datetopia" so we track news and discussions about us. Everytime we post something on Datetopia Blog - Dating Software for Online Services we receive an email a couple of hours or even minutes about the new post. Google really likes blogs, new content and online speeding

dbramley · #12 (**permalink**) 01-30-2008, 03:43 AM

I hear that Yahoo! is the same, particularly if you add your rss feed to its newsreader service.

sgsmorgan · #13 (**permalink**) 01-30-2008, 10:30 AM

Quote:

Originally Posted by dbramley

I hear that Yahoo! is the same, particularly if you add your rss feed to its newsreader service.

How do you add your RSS Feed to the Yahoo Newsreader service?

Cheers

Steve

Orion · #14 (**permalink**) 01-30-2008, 01:43 PM

You can add your blog feed in the Yahoo Site Explorer.They accept urllist.txt, blog feeds and mobile feeds (xhtml or wml).

RawFunk · #15 (**permalink**) 01-31-2008, 08:56 AM

Hi all,

I'm someone who you'd probably call old school. I began my web development career back in 96. You'd mention the web & most people (here is Australia) assumed you were talking about spiders, I kid you not.

Anyway, I was a bit slow on the uptake of blogging, but in the last 6 months I started using it.

Blogs as far as I'm concerned are fantastic, wish I'd taken advantage of them earlier.

To give an example I started a new blog 24 hours ago and it's already indexed in google & achieving top 10 SERPS.

My other blogs get crawled daily, though I do try to post to them at least 3 times a week.

I don't use an automated ping, prefer to do it semi/auto. Still I think the important factor to remember is to write good content.

Although blogs IMHO are a good way to promote single topics addressed in a main web site, I feel just posting a blog topic for the sake of posting is not always going to achieve the desired results.

I'll leave it there for now as I've probably taken this thread out of topic...

Cheers

Dave

uberanimal · #16 (**permalink**) 01-31-2008, 02:21 PM

I just noticed it is added to yahoo and msn just as fast. and yes, he did have a bunch of services list for notification of new posts setting on wordpress. i will have to do this myself. trying it out now and it is working for google.

ushuiah · #17 (**permalink**) 01-31-2008, 02:49 PM

That also happen to my site once. I made a post and ping it. After 30 minutes, I checked it and wow my page was already crawled and indexed.

uberanimal · #18 (**permalink**) 01-31-2008, 03:19 PM

now i just checked another site with my friend and he made a post about a week ago and it is still not indexed. we are getting heavy into researching this issue. looks like google "picks" certain sites for doing a cache of a page/post. so now on to indexing vs. cacheing a page.

Webnauts · #19 (**permalink**) 02-06-2008, 09:56 PM

Quote:

Originally Posted by crankydave

Google picks up the feed. If you do a "blog" search you'll see that the blog posts get picked up immediately. They like to add this to their regular results rather quickly ("minty fresh") and then go back and actually crawl and index the page later.

Dave

But Google doesn't want to crawl RSS feeds: Official Google Webmaster Central Blog: Taking feeds out of our web search results

**crankydave** · #20 (**permalink**) 02-07-2008, 11:43 AM

Quote:

Originally Posted by Webnauts

But Google doesn't want to crawl RSS feeds: Official Google Webmaster Central Blog: Taking feeds out of our web search results

I think you misunderstood my post John.

Google picks up the information from the feed. It's how they know so quickly. Take a snippet of text and the URL, then go back and actually index the page later.

I didn't say they crawled the feed.

Make a blog post. It will show up real fast in their blog search. Within minutes of it being made most of the time and added to their regular results. When it does, go and check the actual post to see if it has been crawled, cached or indexed.

Dave

**wige** · #21 (**permalink**) 02-07-2008, 12:07 PM

Quote:

Originally Posted by Webnauts

But Google doesn't want to crawl RSS feeds: Official Google Webmaster Central Blog: Taking feeds out of our web search results

But Google does crawl feeds. In addition to pingback, this is one of the spider's preferred methods for crawling blogs because the feed contains information in a very standardized form, and the document is generally smaller and easier to parse than a web page. Google does index these pages as well. The cited article onlysays that feeds are filtered so that they do not show up in normal searches. However, Google has a few search utilities that are specifically used for searching their indexed feeds.