Can Bots Index Password Protected Pages ???

[Mike Freemen]

I was wondering if there is a way to allow only a robot to index a certain page that requires a username and password. The Page is also a dynamic page with the following URL

http://grantseekerpro.org/search/Sea...aspx?id=10.001

When people who are not loged in try to go, it takes you to a "Please Login Page" which is fine, but can it index the actual content ?

Do I need to use a robot.txt or what would I do to get that page indexed if it is possible at all?

[adore]

If you want robot to spider this section, you need to provide it with login and password.
I don't know however if it will be sufficient becuase robots reject cookies and your site probably rejects such logging-without-cookies try.

[Mike Freemen]

I see what your saying, I will disable my cookies and see if it will let me login, if it does, how do I give the bot a Username and PW ?

[adore]

If I see correctly, inputs in your login form are named 'TextBox1' and 'TexBox2'.
I was thinking about preparing some url with parameters, however you are using POST method which means that probably values given in url won't be recongized (GET method). You can always try, maybe they will be accepted. The url should look like:
[...]/login.aspx?id=10.001&TextBox1=....&TextBox2=...

You have also some hidden values given, which probably are necessary to be included.

[emi_b]

I don't know the answer to your question, but If you want to make it so that a robot can access that file, doesn't that mean anyone could hack into it, if they figure out the correct URL combination. I think by doing this you might be opening yourself up to an attack.

[sem-seo-pro]

Hi there

Perhaps you are a bit mistaken, robots do not need to login to your webpage. The bots read the coding on your pages from the server.

You can write a disallow statemtent for 1 bot or all bots.

If you do a search on Robots text you can learn how.

Perhaps if you explaim why you want the bot to only read your login page, we can give better insite.

Also having looked at your webpages coding from the one page you seem to have up....the bots wont be spidering you anytime soon or if they do giving it any relevance as you have no keyword or description.

Also as an aside, even if the page was properly optimized it would not have much signifigance to the engines in my eyes.

Also to get Dynamically presented data to be indexed properly by the spiders you would need to do a mod-rewrite in Apache to present static pages of your content to the bots.

Either that or use a CMS that can present Dynamically generated keywords fron the content uploaded to the database, as well as static keywords and descriptions.


Clint

[adore]

Originally Posted by sem-seo-pro

Perhaps you are a bit mistaken, robots do not need to login to your webpage. The bots read the coding on your pages from the server.

They need to make a login in order to index password proteced area.

Originally Posted by sem-seo-pro

Also to get Dynamically presented data to be indexed properly by the spiders you would need to do a mod-rewrite in Apache to present static pages of your content to the bots.

Its just partially true - some robots spider dynamic sites - e.g. Google.

[salubritas]

If you are using server-side scripting (ASP, PHP etc.) for login you can modify the code to allow robots (identified by user-agent or IP address) to access the password-protected pages.

[Conficio]

Excuse me!

Why do you want the search engines to spider the content, if it is private and needs log-in. Does that mean you want many users come to you log-in page? My usual reaction is to turn around immediately!

I find it offensive to publish information in search indexes and then to require a log-in to actually view the information. Several Newspapers do this sort of thing, especially visible in the Google News search. It renders the value of Google's News Search and News Alert of much less value. Again, my reaction and I bet most other's people reaction is to leave immediately. It is not worth to share my personal information, just to get a glimpse on some information. I'd share my info only with some institution, that I seek out because of its good reputation and I want to seek repetitive information from them.

In my opinion you should make a decision, if you want to publish (and index) data or if you want to have private data. Everything else just gets you a lot of angry web surfers.

My five cents worth to the subject.

K<o>

[scisoft]

Originally Posted by sem-seo-pro

Hi there

Perhaps you are a bit mistaken, robots do not need to login to your webpage. The bots read the coding on your pages from the server.

I do not believe this. There are two major ways how to protect files.
Php protection: When I write a php code, I write one set of statement to execute if correct password and login is supplied, the other if it is not. There is absolutely no way how the server can modify the code it is supposed to execute.
Server protection: It is possible to restrict acces to some folders at the server level. But it does not make sense that server will allow robots and dissallow other visitors. Never seen anything similar in the Apache documentation.

[ronniethedodger]

Originally Posted by salubritas

If you are using server-side scripting (ASP, PHP etc.) for login you can modify the code to allow robots (identified by user-agent or IP address) to access the password-protected pages.

I would not go with the user-agent string in this case. It is too easy to spoof that information with a regular browser (such as FireFox) and it will allow non-authorized visitors access to those pages.

IP address checking would be the better choice. But you would have to keep tabs on which IP's the bots are using and keep the list up to date. The IP's move around and adjust very quickly it seems as soon as they are found out, and for good reason...once found out, then a site can serve up alternate content for the spider.

[compusolver]

I've got to take exception to this one:

The bots read the coding on your pages from the server.

As one who has written web robots, I can tell you that bots, like anyone using a browser, can only see the HTML page that your server-side code delivers. They have no special magic to read your PHP, Perl or ASP code.

While I think salubritas had an excellent suggestion for you and I think ronniethedodger gave us some things to think about (but perhaps a little over the top unless you're dealing with monetary matters, or CIA/KGB stuff), in my opinion there is an even better solution -

Why bring users via search engines to a page they cannot access? Why not setup a static page rigged to do well in the SERPS and from there, sell them on signing up for your content? Just a thought.

Good luck!

[Stibbs]

This really seems to be a silly question. I mean protected means protected, not allowed without password!?!
Come on, if you want to lure people to this site do so with static html then all links must be directed to a log in page. ASP and PHP can write a HTML page on the fly during initialization in the global.asa file or similar common file function.