Indexing a Password Protected Site

[jsternfeld]

I've heard that Google and possibly other search engines can index password protected sites. But, I have no idea if there is anything that needs to be implemented on the website in order to make this possible.

I have a client who is considering making their magazine content available via the web, free of charge. But they are convinced that they have to have people subscribe to the website in order to access that content. Like I said it's free of charge and a very simple subscription process, but I don't know how to balance the SEO impact this decision may have. If anyone knows how to do this, I would greatly appreciate any help.

[coder]

add a simple robots.txt file to disallow spiders from indexing your pages

User-agent: *
Disallow: /

[imvain2]

Even though I know you are specifically asking about SEO and password protecting, there was a post on here not to long ago [i forget by whom], but it listed the top 10 things about websites, that turned people away and things that were very annoying about websites. And one of them was forced subscribing to FREE content on the Internet. People generally hate that, and it will lower the amount of people that will want to access the free content.

With that being said, I'm 95% sure that Google and other "legit" search engines won't be hacking your password protected forms and accessing the content. Think about it this way, a search engine would have to run a "brute force" attack on every form it came across to get access to the content behind the form. Its 1. not legal and 2. genreally just not a good idea, even if they were to access that content, they couldn't present that content to users, because the users would have to click on the link in the search engines then have to create an account and login, before they can access the content that the search engines have spidered.

So in short, its not a good idea to force users to subscribe to access free content, and for the legit search engines like Google, MSN and so on, they won't hack your password protected forms.

[compusolver]

If I understand you correctly, you DO want search spiders to index your protected content. That being the case, merely detect the robots in your server-side code (the same script that ensures the user has subscribed) and allow the robots in while forcing others to login.

Your server-side developer will know how to do this, its very straight-forward.

[DrTandem1]

I think you'll need to know the IP address of the spiders to allow them access to the protected directories. However, here is the problem, even if that is possible. If a SE indexes a page in the protected area, the link to such a portion of the site will produce a log-in window. Since the new visitor doesn't know what they are reaching, they will have no way of knowing how to log into the site.

If they execute the log-in window without a valid user name/password, the best you can hope for is a page that tells them they to to register. As someone already mentioned, this is going to be a deal-breaker unless your content is extremely interesting and unique.

[SebastianX]

This idea is half thought out. Search engine users can access the content thru cached copies. Either make it available for everybody, or password protect the whole thingy. Stay away from cloaking as suggested, do not deliver content to spiders and log-in-forms to users.

[Diana]

Rather than cloaking, perhaps you can get users to register/log-in by offering them something NOT available to search engines, an e-book, forum or something else which would require logging in to benefit from.

You might be able to detect engines and serve them your content, but could very well get banned when the SE's detect your cloaking...better to play by the rules.

The good news is offering a site content rich will be great for optimizing. Each page can be optimized (keywords, title) based on the topic of the article. A site rich in content will not have much trouble attracting REAL links to their content & homepage from related sites/pages.

This magazine could very well end up with very high rankings for numerous keywords based on the quality of their content...imagine that...

Diana

[danielle v2.1b]

Originally Posted by compusolver

If I understand you correctly, you DO want search spiders to index your protected content. That being the case, merely detect the robots in your server-side code (the same script that ensures the user has subscribed) and allow the robots in while forcing others to login.

Your server-side developer will know how to do this, its very straight-forward.

This man is absolutely right! Use a user agent script to detect if there is an accepted user agent running or else force session identification. This route bypasses the suggestion that you need SE IP addresses or that visitors can access content displayed in search engines whilst unregistered. It's a simple if/else clause.

I'm really not sure that this is classified as cloaking in the classical sense of the term. Traditionally, cloaking is redirecting SEs to one content and users to another for the purpose of spam however I've seen exactly what compusolve describes employed by several major US educational establishments. I was looking for this particular text and it was in the results. However when I went there it told me I needed to be a resident student.

Maybe if someone knows the difference between an if/else clause based on user agent and cloaking as in the notorious penalty, they can enlighten me?

[DrTandem1]

Let me ask the clarifying question this way:

Why would one want content, that can't be accessed by the visitors at large, indexed? People who have already registered for such content won't be searching for the site (they obviously have already found it) and people who haven't registered won't be able to access it.

I never understood why sites erect roadblocks between the visitor and the goal. If you want them to see free content, why register? If you run an e-commerce site selling widgets, why must they register before they spend their money on your site?

[Webnauts]

Originally Posted by DrTandem1

I never understood why sites erect roadblocks between the visitor and the goal. If you want them to see free content, why register? If you run an e-commerce site selling widgets, why must they register before they spend their money on your site?

For example I am setting up a commercial password protected and rich-content online training program in Web Accessibility, with a total training content of 270 MB. Is that still not fair against the users to your opinion?

[imvain2]

Originally Posted by Webnauts

For example I am setting up a commercial password protected and rich-content online training program in Web Accessibility, with a total training content of 270 MB. Is that still not fair against the users to your opinion?

I agree with DrTandem, however I don't think he meant in every case its not a good idea to password protect content.

[Webnauts]

imvain2 I did not understand, that is why I asked.
I was concerned about this issue myself.

Thanks.

[imvain2]

Originally Posted by Webnauts

imvain2 I did not understand, that is why I asked.
I was concerned about this issue myself.

Thanks.

Oh, I apologize,

I'm used to another website's forum where people are usually being sarcastic.

[imvain2]

Originally Posted by Webnauts

imvain2 I did not understand, that is why I asked.
I was concerned about this issue myself.

Thanks.

IMO, your case is a good case for pw protecting the content.

[Webnauts]

imvain2 no problem. I know that problem. No need to apologize. It could happen that I would think the same. :)

[DrTandem1]

Originally Posted by Webnauts

For example I am setting up a commercial password protected and rich-content online training program in Web Accessibility, with a total training content of 270 MB. Is that still not fair against the users to your opinion?

You're missing the point. I understand that one may want to password protect certain information. However, there is no point in indexing content that can't be accessed. It does makes sense to index content that alludes to more content that may be password protected, but that was not the issue.

The issue was making password protected content search-engine friendly. Well, that is a contradiction. "I want to advertise something that you can't see."

[ADAM Web Design]

Depending on the nature of the content (exclusivity, etc.), there may be a legitimate reason to require registration before content can be viewed. I have a client that requires the user to register to see certain aspects of the site that are otherwise considered confidential. I won't get into who or what, but the idea does have a practical application.

Having said that, this application isn't it. You're requiring registration for what appears to be free content that likely will get cut/pasted by someone into their website without it requiring a registration, thus exposing the content to the world anyway. The SEs caching your pages is also possible.

So...you've got a few options:

1) Present the articles you want to keep exclusive in a format that can't be copied/pasted (e.g. a JPG, GIF, Flash format). As most people will likely say though, this is a stupid idea for bandwidth and SEO purposes (if SEO is a factor.) It's about the only way, though, that you can keep the articles from getting mass-exposed to the public.

2) Give people reasons to register besides just the articles (exclusive offers, giveaways, networking opportunities, whatever you can think of and that preferably hasn't already been done to death like the offers).

3) Skip the registration idea entirely and make it all free. This would be the best idea from an SEO and from a user standpoint, in that anything can be indexed and read for free.

The ONLY thing you need to watch in this scenario is the copy/pasting from #1, unless they give your site and the author credit.

As it relates to password-protecting information, there is a tech site (I won't say which one, because I don't want to give away a secret) that requires users to register as paid users to get their questions asked/answered. And as paid users, they have to log in.

Buuuuuuuut...they goofed soemwhere, because a certain SE caches the actual Q&A pages and thereby makes them public to any savvy user who can get at the cached page. So this doesn't always work either.

[khurramali]

If your reason to password protect the Free content is to collect email addresses of the visitors to your website then just setup a mailing list.

place the subscribe box where ever the free content is on your website.

This way, the vistors will not be forced into giving their personal details, but if they have found useful information and they want to come back whenever you update the information, they can just sign up in your mailing list.

[DrTandem1]

Originally Posted by khurramali

If your reason to password protect the Free content is to collect email addresses of the visitors to your website then just setup a mailing list.

place the subscribe box where ever the free content is on your website.

This way, the vistors will not be forced into giving their personal details, but if they have found useful information and they want to come back whenever you update the information, they can just sign up in your mailing list.

I agree.

[danielle v2.1b]

It seems here that this part of a much bigger, wider discussion. As publishers, we all need to cover the costs of running our site and make some on top for our efforts and so the cost of retention is critical. I guess the notion of allowing SEs to index content that requires registration is really the old dangling the bait trick and it does seem that overall less sites employ that these days. In answer to the original question, yes it entirely possible, and as a designer yours may not be to question why. However I do take dr tandem's point that is actually the best way of doing things?

Take this site for example - there is tons of content you can access! And the big thrust of that content is to draw people in and get them involved and judging by the number of members it seems to be working!

Similarly I use a women's site that posts a lot of thought provoking content that compells you to sign up to get on the Member's board or to chat with other users. In fact when you do sign up, you get access to extra content too, as well as features.

A third site I know is run by a friend of mine who is gay. You have to sign up to see anything because he doesn't want timewasters or to offend accidental visitors. The site allows people to talk and meet and his sign up rate is still pretty high despite nothing visible other than the lure of finding out what's inside.

It seems to me here that the great turn off for people these days in being marketed to. Yet the three sites I mention all use slightly different models from nothing without sign up through to free content that inspires sign up. I think the thing they all do well is provide some social interaction of one form or another based around their content

[kgun]

Jsternfeld wrote:

I've heard that Google and possibly other search engines can index password protected sites. But, I have no idea if there is anything that needs to be implemented on the website in order to make this possible.

I have a client who is considering making their magazine content available via the web, free of charge. But they are convinced that they have to have people subscribe to the website in order to access that content. Like I said it's free of charge and a very simple subscription process, but I don't know how to balance the SEO impact this decision may have. If anyone knows how to do this, I would greatly appreciate any help.

Has the time come to be more precise? The answers you get depends on how accurate your questions are.

1. Do you need https, or
2. http?
3. What do you want the SE Bots to do?
4. What about bad bots?

[zbatia]

khurramali wrote:
-----------
If your reason to password protect the Free content is to collect email addresses of the visitors to your website then just setup a mailing list.

place the subscribe box where ever the free content is on your website.

This way, the vistors will not be forced into giving their personal details, but if they have found useful information and they want to come back whenever you update the information, they can just sign up in your mailing list.
----------
Hm-m-m, I like this idea, and I'll implement it for our Index to let people know when it's updated with new links.

[Webnauts]

Originally Posted by khurramali

If your reason to password protect the Free content is to collect email addresses of the visitors to your website then just setup a mailing list.

place the subscribe box where ever the free content is on your website.

This way, the vistors will not be forced into giving their personal details, but if they have found useful information and they want to come back whenever you update the information, they can just sign up in your mailing list.

I agree too.

By the way: Cool thread!

[wsmeyer]

How about having the first couple of paragraphs of each magazine article freely accessed and then a box to login / register to read the rest of the article.

This would solve your indexing problem. Also, like most people, I hate registering to access content. But if I can read the beginning of it and determine it is what I am looking for, odds are I will register.

William.

[wednesday]

There are many legit uses of cloaking. As example google is ip cloaking and redirects users to country specific pages.

Google has made exceptions to its cloaking rule in the past. For example, its crawler views password-protected content in the Google Scholar program that cannot be viewed by broad Web users, and indexes National Public Radio audio transcripts that are unavailable to Web users, Sullivan said. http://www.eweek.com/article2/0,1895,1774677,00.asp

I couldn't find any trustful information on the subject but you can make your research.

What concerns the technology you can set nocache and your pages should not have visible cache.

[brian.mark]

It's really pretty simple.

You would have your script / server loog for a login. If no login is found, show them an abbreviated version of the page. If a login is found, or if you detect googlebot, slurp, msnbot or another trusted bot, assume that they've logged in. This can be done via .htaccess in Apache or the similar methods in IIS, as well as the possibility of testing within a script that delivers the content. Just look at the HTTP_USER_AGENT environment variable.

As long as people can go to that URL and get that content, even if it's after a login, then you're usually fine. You're also normally fine if you don't use the nocache meta. That implies to the bots that you want people to be able to read the content and aren't trying to hide something.

Brian.