Google ignores robot block

[jetskiron]

I placed the following robot blocker at the top of my header code on 2-17-08 and one day later, Google crawled the website anyway:
<META NAME="ROBOTS" CONTENT="noindex,nofollow">

I'm also wondering if my new website is going to be penalized by google now since this new website is exactly the same as as my existing website under a different domain name? I don't have all my content uploaded to the new website yet. I was going to redirect my old domain name to the new one after I got the new website completely installed, but Google jumped the gun. I did not submit the new domain name to Google either. I suspect the host on the new website, Godaddy, may have alerted Google?

I set up a new hosting account for the new domain name so I could have full control of the hosting account. My existing website is under the control of my web developers group hosting plan, but my web developer just went out of business.

thanks,
Ron

[subhzash]

Originally Posted by jetskiron

I placed the following robot blocker at the top of my header code on 2-17-08 and one day later, Google crawled the website anyway:
<META NAME="ROBOTS" CONTENT="noindex,nofollow">

I'm also wondering if my new website is going to be penalized by google now since this new website is exactly the same as as my existing website under a different domain name? I don't have all my content uploaded to the new website yet. I was going to redirect my old domain name to the new one after I got the new website completely installed, but Google jumped the gun. I did not submit the new domain name to Google either. I suspect the host on the new website, Godaddy, may have alerted Google?

I set up a new hosting account for the new domain name so I could have full control of the hosting account. My existing website is under the control of my web developers group hosting plan, but my web developer just went out of business.

thanks,
Ron

Instead of using robots in meta tags use the robot.txt file. That is useful for you


Regards
Subhash

[jetskiron]

I found the robotstxt.org and it was real simple.

But Google has already indexed my pages from the new site. Whats going to happen? Will they stop showing my pages after they do their next crawl and see the robots.txt file?

thanks much,
Ron

[activeco]

Q: “When I change a robots.txt to exclude more existing files from being crawled, how long does it take for them to be removed from the index? Perhaps the answer is a function of how often the site is crawled and it’s PR?”

A: It is a function of how often the site is crawled. I believe in the past that every several hundred page fetches or several days, the bot would re-check the robots.txt. Note that for supplemental results, you need recrawling to happen by the supplemental Googlebot in order for the robots.txt file to take affect on those pages. If you’re really sure you never want those pages to be seen, you can use our url removal tool to remove urls for six months at a time. But I’d be very careful with the url removal tool unless you’re an expert. If you make a mistake and (for example) remove your entire site, that’s your responsibility. Google can sometimes clear out self-removals, but we don’t guarantee it.

Taken rom muttcutts.com.

[Jean-Luc]

Sorry, but the problem comes from the fact that you didn't do things as they had to.

If the content of the new site is the same as the content of the old web site:
1. upload all contents to the new site
2. put a 301 redirect from every old page to the matching new page (assuming your old host allows it)
3. that's all.

No need for robots.txt or "noindex;nofollow".

Jean-Luc

[activeco]

Jean-Luc,

As I understood he couldn't have any control of the old server and he's putting the same content on a new server and for some reason under a new domain name.

Jetskiron,
unless your site was banned there was no reason for a new domain. Just change dns settings at your registrar.

[incrediblehelp]

Originally Posted by activeco

Jean-Luc,

As I understood he couldn't have any control of the old server and he's putting the same content on a new server and for some reason under a new domain name.

Jetskiron,
unless your site was banned there was no reason for a new domain. Just change dns settings at your registrar.

I disagree. It sounds like he was fully aware that he needed to 301 redirect the old website and should have immediately after the new one went online. I dont see any need to block spiders here.

Anyways , just get the new website up and live with your content fully and put the 301 redirection in place. ofc ourse I am wondering like the others what the reason to make such a change is.

[davidweb]

Originally Posted by jetskiron

I placed the following robot blocker at the top of my header code on 2-17-08 and one day later, Google crawled the website anyway:
<META NAME="ROBOTS" CONTENT="noindex,nofollow">

I'm also wondering if my new website is going to be penalized by google now since this new website is exactly the same as as my existing website under a different domain name? I don't have all my content uploaded to the new website yet. I was going to redirect my old domain name to the new one after I got the new website completely installed, but Google jumped the gun. I did not submit the new domain name to Google either. I suspect the host on the new website, Godaddy, may have alerted Google?

I set up a new hosting account for the new domain name so I could have full control of the hosting account. My existing website is under the control of my web developers group hosting plan, but my web developer just went out of business.

thanks,
Ron

Google always follow its own rule. Google places restrictions for website owners, and surely not for himself.

Google can crawl almost anything from your website ( regardless of whether it is nofollow links, or robots blocking. Google will take your data and silently add into their database. If anything goes wrong ( like some adult link added for example), then google will use blocked data to pass their own judgement.

[Conficio]

DON'T PANIC!

I think you have to better understand how a web crawler like Googlebot works. It gets a huge list of pages to crawl and follows it religiously. When you put a meta tag on your page (or pages) then Google does not stop crawling immediately if it already has the URL's in its list. Later it will analyze the pages read in the crawl and eliminate the URLs from its crawl list, as to your instructions, and also see which pages it should add to the index.

Remember, Google needed to read you page in order to even discover your instructions in the meta-tag.

All what happened so far is that the Google Bot did crawl a few pages. That in itself is not evidence that it will include it in the index (prevented by "NOINDEX") or does follow the links from this page for further crawls (prevented by "NOFOLLOW"). also it takes weeks for Google to purge pages from its index tagged by NOINDEX.

Actually, Google will come back to crawl pages that have links to it from the outside world. And there are some automatic links from sites like About US, which makes a wiki page for every domain registered.

To accelerate the purging of your pages from the Index, sign up for a Google Webmaster account and use Tools --> Remove URLs. Unfortunately ,for that to work you have to first authenticate your site.

Good luck

K<o>

[PAUL-B]

Originally Posted by Jean-Luc

Sorry, but the problem comes from the fact that you didn't do things as they had to.

If the content of the new site is the same as the content of the old web site:
1. upload all contents to the new site
2. put a 301 redirect from every old page to the matching new page (assuming your old host allows it)
3. that's all.

No need for robots.txt or "noindex;nofollow".

Jean-Luc

Sound advice.

[newoptimizer]

I have similar issues, and would like to add a question here..

For a few years I've had geocities pages, and in the last year purchased domain names for a couple of the sites..
I found that I would get kicked out of the search engines--just the new Domain name, not the geocities site itself--unless I used a specific Frames Forward that included a "No Frames" Tag...

I haven't had any trouble with having duplicate pages out there, except yahoo has recently kicked out the new Domain names, while the other major search engines give me a rather high ranking...

Should I have any concerns here?

And any suggestions how to remedy future issues with this?

[gridsix]

Originally Posted by newoptimizer

I have similar issues, and would like to add a question here..

For a few years I've had geocities pages, and in the last year purchased domain names for a couple of the sites..
I found that I would get kicked out of the search engines--just the new Domain name, not the geocities site itself--unless I used a specific Frames Forward that included a "No Frames" Tag...

I haven't had any trouble with having duplicate pages out there, except yahoo has recently kicked out the new Domain names, while the other major search engines give me a rather high ranking...

Should I have any concerns here?

And any suggestions how to remedy future issues with this?

Uhhh, I'd start a site off of geocities and begin the slow migration to that domain. In the short term, whatever keeps you in the search results is a good thing.

[newoptimizer]

thanks for your reply...

the sites are non-income producing, so i have them at geocities because its free..

any suggestions where to place them?

and actually, yahoo didn't kick the domain out, just made it unsearchable by keyword after that domain url had been placed on the first page...

[newoptimizer]

i think i will post this as a new thread..