New Posts = Instant Crawl by Google

[uberanimal]

A buddy of mine keeps asking me why is it when he does a new post on his wordpress blog site, which is about a year old now, google will crawl the new post instantly without any hesitation. I was trying to do some research with this because it only happens to certain sites. Why is this? What have you guys found out about this happening? How does google pick which sites do index the post instantly on?

[carbonize]

Possibly his WP generates a sitemap and notifies Google instantly of updates. I know my Serendipity does.

[crankydave]

Google picks up the feed. If you do a "blog" search you'll see that the blog posts get picked up immediately. They like to add this to their regular results rather quickly ("minty fresh") and then go back and actually crawl and index the page later.

Dave

[uberanimal]

Originally Posted by crankydave

Google picks up the feed. If you do a "blog" search you'll see that the blog posts get picked up immediately. They like to add this to their regular results rather quickly ("minty fresh") and then go back and actually crawl and index the page later.

Dave

So google is pulling the feed first now? hmmm.. this is pretty new isn't it?

[chowell]

The WordPress blog has a "pinging" feature built-in that notifies a pinging engine (which notifies Google) immediately when a new post is made.

Your friend's site is merely benefiting from the insight of the WordPress creators SEO knowledge.

There are other plugins you could add to a WordPress blog to automatically create XML sitemaps & to notify Google of changes to them, but it doesn't come installed as a default, so your friend probably isn't using this method.

[bj]

If the Wordpress installation is set up to ping the update services, this creates instant backlinks, and near instant spider activity.

admin/options/writing and scroll to the bottom.

[wige]

Not sure how new it is. I added feeds to my non-blog website at customer request several months ago, and not only does new content get indexed very quickly, but we also perform quite well on Google's blog search. Also, if the blog is set to ping Google when it is updated, that may trigger the addition of the new content almost immediately as well.

[Orion]

Originally Posted by bj

If the Wordpress installation is set up to ping the update services, this creates instant backlinks, and near instant spider activity.

admin/options/writing and scroll to the bottom.

I've had good success with this too.. I have a related question.. does anyone have or know of a fairly comprehensive list of update services that are recommended to subscribe to? I found it quite a time consumer the first couple times searching them all out.. Some are obvious, but I'm sure there's a ton I'm missing.

[bj]

The link to the Wordpress docs that is right in that section in the Wordpress admin interface leads to a list that is kept pretty up to date.

[Orion]

DOH! .. figures got a question check the help files.. ;o) Thanks!

[datetopia]

We have a google news alert for the keyword "datetopia" so we track news and discussions about us. Everytime we post something on Datetopia Blog - Dating Software for Online Services we receive an email a couple of hours or even minutes about the new post. Google really likes blogs, new content and online speeding

[dbramley]

I hear that Yahoo! is the same, particularly if you add your rss feed to its newsreader service.

[sgsmorgan]

Originally Posted by dbramley

I hear that Yahoo! is the same, particularly if you add your rss feed to its newsreader service.

How do you add your RSS Feed to the Yahoo Newsreader service?

Cheers

Steve

[Orion]

You can add your blog feed in the Yahoo Site Explorer.They accept urllist.txt, blog feeds and mobile feeds (xhtml or wml).

[RawFunk]

Hi all,

I'm someone who you'd probably call old school. I began my web development career back in 96. You'd mention the web & most people (here is Australia) assumed you were talking about spiders, I kid you not.

Anyway, I was a bit slow on the uptake of blogging, but in the last 6 months I started using it.

Blogs as far as I'm concerned are fantastic, wish I'd taken advantage of them earlier.

To give an example I started a new blog 24 hours ago and it's already indexed in google & achieving top 10 SERPS.

My other blogs get crawled daily, though I do try to post to them at least 3 times a week.

I don't use an automated ping, prefer to do it semi/auto. Still I think the important factor to remember is to write good content.

Although blogs IMHO are a good way to promote single topics addressed in a main web site, I feel just posting a blog topic for the sake of posting is not always going to achieve the desired results.

I'll leave it there for now as I've probably taken this thread out of topic...


Cheers

Dave

[uberanimal]

I just noticed it is added to yahoo and msn just as fast. and yes, he did have a bunch of services list for notification of new posts setting on wordpress. i will have to do this myself. trying it out now and it is working for google.

[ushuiah]

That also happen to my site once. I made a post and ping it. After 30 minutes, I checked it and wow my page was already crawled and indexed.

[uberanimal]

now i just checked another site with my friend and he made a post about a week ago and it is still not indexed. we are getting heavy into researching this issue. looks like google "picks" certain sites for doing a cache of a page/post. so now on to indexing vs. cacheing a page.

[Webnauts]

Originally Posted by crankydave

Google picks up the feed. If you do a "blog" search you'll see that the blog posts get picked up immediately. They like to add this to their regular results rather quickly ("minty fresh") and then go back and actually crawl and index the page later.

Dave

But Google doesn't want to crawl RSS feeds: Official Google Webmaster Central Blog: Taking feeds out of our web search results

[crankydave]

Originally Posted by Webnauts

But Google doesn't want to crawl RSS feeds: Official Google Webmaster Central Blog: Taking feeds out of our web search results

I think you misunderstood my post John.

Google picks up the information from the feed. It's how they know so quickly. Take a snippet of text and the URL, then go back and actually index the page later.

I didn't say they crawled the feed.

Make a blog post. It will show up real fast in their blog search. Within minutes of it being made most of the time and added to their regular results. When it does, go and check the actual post to see if it has been crawled, cached or indexed.

Dave

[wige]

Originally Posted by Webnauts

But Google doesn't want to crawl RSS feeds: Official Google Webmaster Central Blog: Taking feeds out of our web search results

But Google does crawl feeds. In addition to pingback, this is one of the spider's preferred methods for crawling blogs because the feed contains information in a very standardized form, and the document is generally smaller and easier to parse than a web page. Google does index these pages as well. The cited article onlysays that feeds are filtered so that they do not show up in normal searches. However, Google has a few search utilities that are specifically used for searching their indexed feeds.

[Webnauts]

Originally Posted by crankydave

I think you misunderstood my post John.

Google picks up the information from the feed. It's how they know so quickly. Take a snippet of text and the URL, then go back and actually index the page later.

I didn't say they crawled the feed.

Make a blog post. It will show up real fast in their blog search. Within minutes of it being made most of the time and added to their regular results. When it does, go and check the actual post to see if it has been crawled, cached or indexed.

Dave

Google accepts pings to their blog search service.

For instance, with Pingomatic, WordPress users can rest easy that their posts will show up quickly on Google Blog Search!

As I explained above, your RSS feeds must not be accessed by Googlebot. That is a general blogs practice already.

How can Google pick up the information from a feed since it is disallowed in the robots.txt?

Here are instructions how to, directly from Google: Google Blog Search Help

If your blog is not indexed already, you can submit it here: Google Blog Search Ping Service

[kgun]

Originally Posted by uberanimal

A buddy of mine keeps asking me why is it when he does a new post on his wordpress blog site, which is about a year old now, google will crawl the new post instantly without any hesitation. I was trying to do some research with this because it only happens to certain sites. Why is this? What have you guys found out about this happening? How does google pick which sites do index the post instantly on?

I have noted the following:

1. Google index a WPW post while I edit it.
2. Worst or best for me and WPW, some of them get top SERP position.

[kgun]

Originally Posted by bj

If the Wordpress installation is set up to ping the update services, this creates instant backlinks, and near instant spider activity.

admin/options/writing and scroll to the bottom.

Easy Java Script coding:

Build*Your*Own AJAX Web*Applications - SitePoint Books One of the first code examples.

[Webnauts]

Originally Posted by kgun

Easy Java Script coding:

Build*Your*Own AJAX Web*Applications - SitePoint Books One of the first code examples.

I think what Bj mentioned was used before Pingomatic function have been implemented:

http://blogsearch.google.com/ping/RPC2″ to the “Update Services” in your WordPress Dashboard under Options->Writing.

What does that book have to do with this topic? I do not understand.

[kgun]

Code to ping a site that can be directly reused on your own and it is exellent as a book on soft introduction to OOP. In addition you learn AJAX programming. I can not give a better introductary reference to OO web developement.

[crankydave]

Originally Posted by Webnauts

Google accepts pings to their blog search service.

For instance, with Pingomatic, WordPress users can rest easy that their posts will show up quickly on Google Blog Search!

As I explained above, your RSS feeds must not be accessed by Googlebot. That is a general blogs practice already.

How can Google pick up the information from a feed since it is disallowed in the robots.txt?

Here are instructions how to, directly from Google: Google Blog Search Help

If your blog is not indexed already, you can submit it here: Google Blog Search Ping Service

blog rss feed - Google Blog Search

Dave

[Webnauts]

Originally Posted by crankydave

blog rss feed - Google Blog Search

Dave

I could only see there .html pages and not XML format feeds.

Never matter.

[kgun]

John, depends on how you sort.

I sorted by past month and this

The Alagad Technical Blog Rss Feed

was hit number 3.

The article has a story:

"Know Any Good SEO and Web Marketing Contractors"? that should be relevant for you.

What a different view if you click the above link in IE 6.X and in modern browsers.

I sometimes like to see the source directly, so I.E. 6.x is great for some tasks.

[jordanmcclements]

I concur about blogspot.

New posts there get indexed very quickly for me...

[Webnauts]

Have you all already met the Google fresh content blog penalty? http://www.hobo-web.co.uk/seo-blog/index.php/fresh-content-penalty/

[kgun]

What about my blog without fresh information in two years? Desperately writing fresh content each day. That is another business.

Nevertheless the Blog post cited in the thread below is IMO still very actual and relevant today and it goes to the heart of any searh engine.

Did The Recent TrustRank Patent Grant Murder DMOZ?

You need to know the weakness and limitation of a computer alogrithme and more precisely a BOT. PageRank, TrustRank and "LobbyRank" don't trust anybody ...

Uswe common sense and read between the lines.

[innominds]

Wordpress pings the blogposts with the help of pingomatic. So it may a reason to get the instant crawl by google.