Tramendous daily Bot Activity, But not indexed

bbott · #1 (**permalink**) 10-24-2004, 09:58 PM

About 2 weeks ago our company launched the following forum:

Webmaster Forum

... Numerous times during the day we get very heavy bot activity on all forum pages - BUT not a single forum page is indexed yet.

We have also removed the SID from the URLs and also made our URLs search engine friendly.

For example previous URLs would appear like this:

http://ewebdevelopment.net/forum/viewtopic.php?t=36

Now the URLs appear like this:

http://ewebdevelopment.net/forum/google-forum-vf2.html

Anybody have any idea why none of our forum pages are being indexed yet? BOTs are chewing up our bandwidth like no tomorrow!!

randfish · #2 (**permalink**) 10-24-2004, 10:05 PM

I'm unfamiliar with this scenario - can you tell us which bots have been visiting - do you have IPs?

There are some posters around who are very familiar with all of the types of bots and individual crawlers and could help you to ID them and let you know whether your situation is unique.

bbott · #3 (**permalink**) 10-25-2004, 01:36 PM

Is this a googlebot?

66.249.65.240

This one has been eating my site alive by the minute.

DriWashSolutions · #4 (**permalink**) 10-25-2004, 03:19 PM

Quote:

Originally Posted by bbott

Is this a googlebot?

66.249.65.240

This one has been eating my site alive by the minute.

Yep, it is.

IP's that begin with 66.249 are usually GBots. G has a whole slew of IP's beginning with that. Check it out here:

http://www.whois.sc/66.249.65.240

greeneagle · #5 (**permalink**) 10-25-2004, 04:56 PM

Is that a 2 week old "sandbox" virgin you are asking about?

IMO - That's almost pedophilic!

Ken

bbott · #6 (**permalink**) 10-25-2004, 05:13 PM

Quote:

Originally Posted by greeneagle

Is that a 2 week old "sandbox" virgin you are asking about?

IMO - That's almost pedophilic!

Ken

My domain is no longer sandboxed. The only new thing is the /forum url.

Also, even if it was sandboxed doesn't mean google wouldn't index my forum pages.

greeneagle · #7 (**permalink**) 10-25-2004, 05:15 PM

The link you provide for the forum does not have the "www" out front. If that is what you have been using Google may see it as "virgin" indeed! If that's the case jerk her out of the sandbox with a full URL!

Ken

bbott · #8 (**permalink**) 10-31-2004, 04:09 PM

Quote:

Originally Posted by greeneagle

The link you provide for the forum does not have the "www" out front. If that is what you have been using Google may see it as "virgin" indeed! If that's the case jerk her out of the sandbox with a full URL!

Ken

Wups, didn't notice that.

I have been linking to it w/ a full URL though...

Infact many of our forum pages are now indexed, so i gues i was just a little too impatient :)