Website never indexed in Google after 5 months

[Bolamega]

I hope this is the right place for this question. Sorry if it's not.

I started a site last September, and the Googlebots visit regularly, but it has never been indexed in Google. It has, however, been assigned a pagerank of 4, so this is really bizarre to me. Also, Google Webmasters tools reports that it's ranking around 187 for the term "apartment", but I can find it anywhere. I'm so confused!

The site is http://bohemianrevolution.com, and I'm wondering if anyone can look at it and see something I'm missing that explains it not getting indexed.

I've started about a dozen similar sites - all WordPress blogs. Even with a very slow start on my part, they all get indexed by Google within days - except this one. The one thing that's really different about this domain from the others is that it was owned by someone else from 2001 to 2004 (I think), and then it lapsed, and then I bought it in 2005. Could that explain this problem?

I'm wondering whether I should just wait and assume Google will eventually index, or start with a new domain, point this one to it, and import the posts. But if it's not just the fact of the domain having existed before (and been indexed, when it was the old site), then whatever mistake I'm making could just follow me to the new domain.

I have no idea, really. I don't even do much in the way of SEO - I just rely on the blog structure and blogrolls to do that for me, and then I follow the basic rules of creating a good site. I can't imagine what sort of "black hat" rule I could've broken. I'd appreciate any feedback.

[Webnauts]

I guess you need quality backlinks. To get things settled faster, submit your blog feed to feed directories.

[Bolamega]

I don't think that's it. I've started blogs that ended up sitting for months without posts as I got sidetracked: their home pages were indexed within days of launch, with no inbounds and no site activity.

Besides, this site has a fair amount of inbounds. Check it out on Yahoo and MSN, who indexed it within days, like every other blog.

The inbounds don't determine WHETHER you'll be indexed, they determine how well you'll be indexed. And again, it's had PR4 for over a month, so why does it have decent PR if it's banned from the index?

And I do think this one is banned, and I can't figure out why, or whether I should just hang in there, or what.

[Webnauts]

Are we talking about Google or Yahoo and MSN?
How many quality links do you have with Google?

[Bolamega]

Currenltly, Google displays one link. But a couple of months ago, it showed 60, and it didn't index the site, either.

Again, Google always indexes blogs immediately, with or without inbounds. It doesn't rank them well without quality inbounds, but it does index them.

[Webnauts]

Originally Posted by Bolamega

Again, Google always indexes blogs immediately, with or without inbounds. It doesn't rank them well without quality inbounds, but it does index them.

I think you are not aware about the ongoing Google updates since last December. But anyway, I do not want to continue this discussion, especially when someone doubts about what I am saying. Sorry.

I hope someone else can help you out. I can't.

[Bolamega]

No, I am quite aware of Google's updates. BUT I started another blog in October of last year. It currently has 4 posts in all those months, and no inbounds. It was indexed within a week. How, then, can inbounds be the issue?

There is definitely something "wrong" with this domain according to Google's algo, and the only thing I can see that distinguishes it from my sites that are getting indexed is that it used to be owned by someone else.

I hope someone reads this who does not assume I don't know what I'm talking about when I assure you I have exactly similar blogs started around the same time which were indexed within days of launch.

[Webnauts]

Could it be that Google could not crawl properly your blog because of its invalid HTML code (209 errors)? http://validator.w3.org/check?uri=ht...olution.com%2F

[Hiops]

I just checked and the site has 66 IBL on Google and 45 pages indexed. It's doing OK, but has 209 errors.

[Webnauts]

Originally Posted by Hiops

I just checked and the site has 66 IBL on Google and 45 pages indexed.

Alex where did you find those 66 IBLs?

[Hiops]

On google right here:
http://en-us.start2.mozilla.com/sear...2F&btnG=Search
and one of the links right from this very thread.

[Webnauts]

Is this your robots.txt?

#
# WebmasterWorld.com: robots.txt
# GNU Robots.txt Feel free to use with credit given to WebmasterWorld.
# Please, we do NOT allow nonauthorized robots any longer.
# http://www.searchengineworld.com/robots/
# Yes, feel free to copy and use the following.

User-agent: Nutch
User-agent: Jetbot/1.0
User-agent: Jetbot
User-agent: WebVac
User-agent: Stanford
User-agent: Stanford CompSciClub
User-agent: Stanford CompClub
User-agent: Stanford Spiderboys
User-agent: scooter
User-agent: naver
User-agent: dumbot
User-agent: Hatena Antenna
User-agent: grub-client
User-agent: grub
User-agent: WebZip
User-agent: larbin
User-agent: b2w/0.1
User-agent: psbot
User-agent: Python-urllib
User-agent: URL_Spider_Pro
User-agent: CherryPicker
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: WebBandit
User-agent: EmailWolf
User-agent: ExtractorPro
User-agent: CopyRightCheck
User-agent: Crescent
User-agent: SiteSnagger
User-agent: ProWebWalker
User-agent: CheeseBot
User-agent: LNSpiderguy
User-agent: Mozilla
User-agent: mozilla
User-agent: mozilla/3
User-agent: mozilla/4
User-agent: mozilla/5
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
User-agent: Teleport
User-agent: TeleportPro
User-agent: Stanford Comp Sci
User-agent: MIIxpc
User-agent: Telesoft
User-agent: Website Quester
User-agent: moget/2.1
User-agent: WebZip/4.0
User-agent: WebStripper
User-agent: WebSauger
User-agent: WebCopier
User-agent: NetAnts
User-agent: Mister PiX
User-agent: WebAuto
User-agent: TheNomad
User-agent: WWW-Collector-E
User-agent: RMA
User-agent: libWeb/clsHTTP
User-agent: asterias
User-agent: httplib
User-agent: turingos
User-agent: spanner
User-agent: InfoNaviRobot
User-agent: Harvest/1.5
User-agent: Bullseye/1.0
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: CherryPickerSE/1.0
User-agent: CherryPickerElite/1.0
User-agent: WebBandit/3.50
User-agent: NICErsPRO
User-agent: Microsoft URL Control - 5.01.4511
User-agent: DittoSpyder
User-agent: Foobot
User-agent: WebmasterWorldForumBot
User-agent: SpankBot
User-agent: BotALot
User-agent: lwp-trivial/1.34
User-agent: lwp-trivial
User-agent: BunnySlippers
User-agent: Microsoft URL Control - 6.00.8169
User-agent: URLy Warning
User-agent: Wget/1.9
User-agent: Wget/1.6
User-agent: Wget/1.5.3
User-agent: Wget
User-agent: LinkWalker
User-agent: cosmos
User-agent: moget
User-agent: hloader
User-agent: humanlinks
User-agent: LinkextractorPro
User-agent: Offline Explorer
User-agent: Mata Hari
User-agent: LexiBot
User-agent: Web Image Collector
User-agent: The Intraformant
User-agent: True_Robot/1.0
User-agent: True_Robot
User-agent: BlowFish/1.0
User-agent: JennyBot
User-agent: MIIxpc/4.2
User-agent: BuiltBotTough
User-agent: ProPowerBot/2.14
User-agent: BackDoorBot/1.0
User-agent: toCrawl/UrlDispatcher
User-agent: WebEnhancer
User-agent: suzuran
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: VCI
User-agent: Szukacz/1.4
User-agent: QueryN Metasearch
User-agent: Openfind data gathere
User-agent: Openfind
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Xenu's
User-agent: Zeus
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: RepoMonkey
User-agent: Microsoft URL Control
User-agent: Openbot
User-agent: URL Control
User-agent: Zeus Link Scout
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Webster Pro
User-agent: EroCrawler
User-agent: LinkScan/8.1a Unix
User-agent: Keyword Density/0.9
User-agent: Kenjin Spider
User-agent: Iron33/1.0.2
User-agent: Bookmark search tool
User-agent: GetRight/4.2
User-agent: FairAd Client
User-agent: Gaisbot
User-agent: Aqua_Products
User-agent: Radiation Retriever 1.1
User-agent: WebmasterWorld Extractor
User-agent: Flaming AttackBot
User-agent: Oracle Ultra Search
User-agent: MSIECrawler
User-agent: PerMan
User-agent: searchpreview
User-agent: sootle
User-agent: es
User-agent: Enterprise_Search/1.0
User-agent: Enterprise_Search
Disallow: /

User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/
-----------------------------------------

You disallow all robots to crawl and index your site!!! See above the rule you use, large and in red! And get rid of it now!

LOL

[Bolamega]

(1) The HTML errors are a recent issue due to some posts I imported that I'm slowly re-coding. But the refusal to index was going on for 4.5 months before that.

(2) According to my research, that Disallow: / code in that position tells only the bots above it to ignore the site. I've changed it anyway, just in case. But I'm not convinced that's the problem because it's been up on several of my sites sites for a month now, and their new pages were getting crawled regularly. Plus, that doesn't explain the first four months of Google refusing to index the site for 4 months. I'll let you know if it works, though.

(3) Hiops, when I ran your link, the search term had a space between "link:" and "bohemianrevolution.com". Without the space, I get one link. However, under no conditions - including a McDar datacenter check - am I seeing any pages indexed, as you reported. And yet Google Webmasters says 0 pages indexed, but that I'm ranking for the term "apartment" and have PR4.

[Webnauts]

Originally Posted by Bolamega

(2) I've removed the Disallow: / code, BUT I'm not convinced that's the problem because it's been up on several of my sites sites for a month now, and their new pages were getting crawled regularly. Plus, that doesn't explain the first four months of Google refusing to index the site for 4 months. I'll let you know if it works, though.

I am very sure that you have added the Disallow: / rule after your site have been indexed. And be happy that your site is not excluded yet from the search engines.

I am surprised though that someone here in our community has doubts about my competencies in SEO, and especially for a bloody issue like that, which even a newbie could tell that what you did is a disaster.

Anyway, if you need any help again, I am sure others here will be glad to help. ;)

And besides, I would like to introduce myself:
http://www.webproworld.com/viewtopic.php?p=350066

[Bolamega]

No, the site was never indexed. I checked almost every single day of the 5 months. It wasn't indexed before the robots.txt change, and it wasn't indexed after, either.

And none of the other sites running that robots.txt were de-indexed. In fact, they all continued to get new pages indexed.

Just because I don't immediately accept the first suggestion you make, and provide evidence as to why that suggestion is not the cause, does not mean I think you're incompetent. It means I think you were being dismissive. ;)

[incrediblehelp]

Webnauts (John) is right:

This example keeps all robots out:

User-agent: *
Disallow: /

Get rid of the foward slash.

[Hiops]

Hi, John! I just checked his robots.txt and he disallowed almost all SE possible( btw, why he did that?) , except GoogleBot, and I think this is why he has some links on Google. Would you please correct me, if I'm wrong.

[Webnauts]

Originally Posted by Hiops

Hi, John! I just checked his robots.txt and he disallowed almost all SE possible( btw, why he did that?) , except GoogleBot, and I think this is why he has some links on Google. Would you please correct me, if I'm wrong.

I can only tell that the site have been indexed before he added with copy and paste that ancient robots.txt file.

[Hiops]

It's the most logical explanation to me, why they added this stupid robot.txt file there, just nobody knows.