Following a hunch -- MSN and robot tags

[jawn_tech]

Disclaimer -- this is totally unscientific. Just one site, just one instance, with just 3 keywords.

A particular site I manage does well in the SE's, and has been #1 on MSN for 3 important keywords.

Full knowing the major SE's will index a site even without a
<META NAME="ROBOTS" CONTENT="ALL"> or, a...
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">

But hey, who knows if there's some search engine out there that it may help in, and a few extra visitors every now and then certainly won't hurt. After all, just because the majors don't use them (unless you tell them "noindex,nofollow", which even then is debatable sometimes), they certainly wouldn't knock you for using the "index,follow"...or would they?

So I go ahead and throw it in, just to see if it makes any difference in traffic/referrals.

In a couple days the 3 keywords getting first place in MSN suddenly go to page 2, in the 11-20th range. Hmm. So I remove the tags, and in a week they're back to number one. It's the only SE where I noticed a difference.

Just a coincidence? Probably, but the timing sure looked fishy, since no other changes were made, as far as code, content, or IBL's (that I was aware of). Would be interesting to run a test with a number of different sites.

[timmathews.com]

Wow. Nice hypothesis. I will run web pos report for MSN on one of my sites, then remove tags and run again in 14 days. I will let you know results.
Tim

[jawn_tech]

Thanks! MSN still seems like a new breed of animal, and it's interesting to test its temperment by throwing a stick at it once in a while.

[bblwise]

About the time that MSN introduced it's new search, my website dropped out completely from results. I've deleted my robot.txt code and will see what happens.

It was fine under the old sea.search

[webmasterjunkie]

I use the:
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
on websites. MSN places me well in the first page for a term I shoot for. One site has this code and no robots.txt file. Just thought I'd share.

[montytx]

I am suspicious that a change in code seems to move sites up in rankings. Almost as if MSN thinks that a fresher website is worth more than one with no changes. My site has yoyoed from no to 17 to 5 last week to 10 yesterday to 18 today. The only correlation I can find is that when I change the content of the page I notice a temporary boost in rankings. The problem is then 5 days later they seem to drop. Has anyone else noticed an similar occurance?

[webmasterjunkie]

All search engines will favor a site with fresher content than a website with stale content. But, they can also tell the difference from a website that needs fresh content from a website that has no need for fresh content.

IMO, it's a matter of what you are trying to get across to the user.

[garbageaccount]

I'll try it for a few weeks and see what happens. It really can't hurt at this point.

[wrmineo]

It's very possible you're on to something, but it's just as likely you're dealing with coincidence rather than conclusion.

I place these tags in all of my main pages and have only seen good results in MSN. Case in point, my main page at http://www.britney.freshdames.com has these metas and when I query MSN for "britney spears pics" my page is #4 out of nearly 3 Million results.

In all honesty, I'm not willing to wager that removing them will propel the page up a notch or two.

However, I'm happy to participate in a little curiousity and research. I'll take an unranked page or two and tweak them and let you know.

Good luck - keep the notes and information flowing.

Happy Surfing ~

[yeshua14]

Tim,

After the 14 days, did anything happen?

[garbageaccount]

Well mixed reviews here.

While the indexing rate for our sites on MSN doubled it plumeted for Google. So I have put in a Google specific tag to see if that helps.

[DMC_34]

I am #4 on MSN with a 4 month old site www.pro-gift-baskets.com The #1 and #2 uses blank metas tags except for a title and has the #2. Mostly due to links though. IMO meta tags are meaningless to MSN, I have seen many #1's on big keywords which do not even use metas. Just my .02

[webhost1]

If a site was to throw more pages into the search engines would this help them in any way?

[cspelts]

In ten years as a web developer I've never once used robots tags. Since the default is index and follow, the tags seemed unnecessary to me. Just my opinion...

[misterc]

Hope you don't mind the comments as I'm not an expert like you guys. Had an e-commerce site created for me by 'experts' NOT! They made all the mistakes they shouldn't so I've had to learn how to optimise etc. The sites are http://www.furthrows.com and http://www.furthrows.co.uk Firstly Ihave noticed this 'robot' ttxt in html and wondered if it helped spiders to pick up level 2 of sites as Google is only just picking up home page on mine. Also How do I submit a url to msn or don't I; do I have to wait for them to find me. I think I have the best products/service and prices but as they say, what's the point in that if nobody knows about it! Any help appreciated.

[webhost1]

Posting on msn. You should be able to find a link on the site some place that says add your site. At least that is how I think I did it.

[srlabs]

Certain directories can get you indexed too, this is useful for search engines that have converted to pay for inclusion only. I remember balking when inkotomi wanted 200 dollars for inclusion, eventually during link building I found a directory that fed listings to Inkotomi SE's. Now I am listed and I get one whole visitor a day from their network, I know if I had paid I would feel ripped off because they generate virtually no traffic.

Oops off topic, I know that now my Overture ads display on MSN as well as Yahoo, are you listed on Yahoo? They may share organic info as well.