robots.txt an open window?

[simonm]

I use robots.txt to identify what I don't want on the search engines. But it struck me after an increase in hack attempts that maybe all that robots.txt was doing was telling the hackers where I didn't want them to go and therefore where they might want to go!

I've ditched robots.txt and just made sure any pages I want off search engine listings have no inbound links from public pages.

Maybe I'm paranoid!

[inertia]

What's your set-up? Are you using a CMS?

Pages can be found by G in a variety of ways, not just from inbound links. Do the pages link out anywhere?

If I want to block a page from the index I use the noindex meta tag.

[Smeagol]

I fear there be more to be done here than just a 'nofollow' alone, but it is good start. Like 'nofollow', noarchive' and 'noindex'....
However. Been there, tried it and did not get the t-shirt. If bad intentions are out to get you, they never follow the rules!
Try putting a "bad bot/Hack" trap as the first exclude in the robots.txt file, and possibly a few other nice tricks to put off the unlawfull access.
And if they do go there.. where you told them not to go, fload then with (what looks like honey) garbage. Knowing the search engines engines advise against this kind of tactic, it has worked wonders for me, and the 'B', "Y" and 'G' has not tried it yet. So look after yourself and your site and not what the search engines advise!

[deepsand]

Originally Posted by simonm

I've ditched robots.txt and just made sure any pages I want off search engine listings have no inbound links from public pages.

Well, unless there are also no internal links pointing to said pages, they're easy discovered.

If there are, all you've accomplished is to implicitly allow the well behaved bots to do as they like.

[Smeagol]

I agree completely with you deepsand. But when you have a private site and want (select informed) others to see it and give comments (private url) and suddenly every search engine is on your site! (exaduration).

For me this means that other Internet users supplied with the URL in question either willingly gave it out or there is foul play?

I vote for foul play.tMany fantastic 'helpers' that plugin into your browser that even (previously) respected software companies force onto users are to blame for this espionage of detection. The result is quickly know. You are idenitfied and tracked.

At least that is how I have experienced it. The search engine 'recomendations' are one way and not to your benifit. As for the rest. I see there is more bad than good out on the internet at present. So deal with it.

However. robots.txt is good it helps the 'good people' to help themselves and everyone else to be moe efficient and respect privacy. It does not take away that they are many more others out there that want to exploit it.

I chose to deal with the prblem from a claisical point of view. 'If you don't listen, then feel'.

[deepsand]

One must bear in mind the the web is a public space, not a private one. And, it is a long ago and well established principle that there can be no reasonable expectation of privacy in a public space.

Therefore, if one is desirous of some modicum of privacy in a public space, it is incumbent on that person to take such steps as are possible to shield that which is to go unseen and unheard by the public at large.

The use of robots directives is one method of undertaking such shielding; this will easily at least handle the case of well behaved bots. For those ill behaved, one might consider IP blocking; while not a perfect solution, it is better than none at all.

As in all things, physical or virtual, the maxim is that, when in public, expect no privacy. For that one should remain in private spaces.

[luigip]

Originally Posted by simonm

I use robots.txt to identify what I don't want on the search engines. But it struck me after an increase in hack attempts

Am I right that these pages are password protected? So these pages are already "protected from the search engines" and there is no need for your robots.txt solution.

[simonm]

Hi, Some good advice here and yes, I'll put the nofollow / noindex on page rather than robots.txt, should have thought of that.
Yes, pages are password protected and unlinked from the outside world and yes, doing a regular trawl of what the search engines list, they do seem to be able to get into every nook and cranny!

[deepsand]

Originally Posted by simonm

... I'll put the nofollow / noindex on page rather than robots.txt ...

If a bot is not so well behaved as to heed robots directives, why believe that it will heed meta tags?

[inertia]

Why would a password protected page be indexed? Bots cant fill in logins. There must be something dodgy going on there!

[moneymen]

U can do basic auth the pages under basic outh unless u give them permision by entering the login for the basic auth in webmaster tools

[deepsand]

Originally Posted by moneymen

U can do basic auth the pages under basic outh unless u give them permision by entering the login for the basic auth in webmaster tools

What has Google Webmaster Tools to do with it?

[moneymen]

I think i saw option somewhere when u are adding your site to put also user and password for password protected things so google bot can crawl it

[deepsand]

Originally Posted by moneymen

I think i saw ...

So, you are guessing?

Citation required.

Even were such the case, would have no effect on SEs other than Google.

Blocking Google

[moneymen]

NO am not guessing i saw it

not sure for other SE i was saying about google although it think same works for other ones also since i had a site that was protected that way "private one" and only public parts was index and 2 other files.After full 2 years of existing nothing exchept those 3 public ones is not indexed
and on the link u provided Blocking Google first
first method mentioned htaccess one is the one am talking about-- basic auth

[deepsand]

Originally Posted by moneymen

I think i saw

Originally Posted by moneymen

NO am not guessing i saw it

Not what you first said. "I think I saw" and "I saw" are two quite different things.

Originally Posted by moneymen

U can do basic auth the pages under basic outh unless u give them permision by entering the login for the basic auth in webmaster tools

Originally Posted by moneymen

... on the link u provided Blocking Google first
first method mentioned htaccess one is the one am talking about-- basic auth

Again, not what you first said. htaccess is not in Webmaster Tools!

It is important that one both precise and sufficiently complete when speaking.

[moneymen]

First there is a part of text missing from my first post i don't know why

the full one was

U can do basic auth of the pages, pages under basic auth can't be crawled unless u give them permision by entering the login for the basic auth in webmaster tools

maybe that is what confused u since i didn't said that u are seting up basic auth in webmaster tools
and later i said that u posted link to a resource where first method explained is the one am talking about basic auth = htaccess password protection = popup login

[deepsand]

Originally Posted by moneymen

U can do basic auth of the pages, pages under basic auth can't be crawled unless u give them permision by entering the login for the basic auth in webmaster tools

maybe that is what confused u since i didn't said that u are seting up basic auth in webmaster tools

To the contrary, that is precisely what you said.

"u give them permision by entering the login for the basic auth in webmaster tools"

One does not do anything "in Webmaster Tools."

Originally Posted by moneymen

... and later i said that u posted link to a resource where first method explained is the one am talking about basic auth = htaccess password protection = popup login

That does not change the fact that you first mis-spoke.

As previously noted, precision of speech is very important.

[moneymen]

Are u an english teacher ?
I think its pretty clear am not explaining to a baby am i ?

And when u take half of the statement from the sentence it gets different meaning so if u quote something quote all mr.eng teacher


p.s. point of the thread was to help him and not to corect things i say to first grade english its clear enough
i don't know what is wrong with some people here they will turn whole subject in different way only so they can get into debate with someone maybe admins should make new section called debating room if people here are so horny on debates

[deepsand]

That you know what you intend to communicate is not sufficient for the understanding of others.

One cannot help when what one says is not clearly stated.

I think its pretty clear am not explaining to a baby am i ?

Having reviewed some of your posts elsewhere as well, it is abundantly clear as to who is being puerile.

[moneymen]

ya because am in hostile environment

[deepsand]

Any appearance of hostility is of your own making.