www.seoworkers.com showing error

[subhzash]

Hi All,

When i want to see the internal links of SEO Workers Search Engine Optimization Consulting Company through dead-links.com and also with XENU
It is giving following error

Dead-links.com shows this error

SEO Workers Search Engine Optimization Consulting Company 403 Forbidden

XENU shows that

No such host



Regards
Subhzash

[kgun]

Probably that implies that that bot is blocked.

[Webnauts]

We deny domain access to spammers, spies and other scumbags with our .htaccess file. Bad bots attacks are reduced up to at least 95% so far!!!

To be specific here is the rule which can be helpful for everyone else too:

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ADSARobot|ah-ha|almaden|aktuelles|Anarchie|amzn_assoc|ASPSeek|ASSORT|ATHENS|Atomz|attach|attache|autoemailspider|BackWeb|Bandit|BatchFTP|bdfetch|big.brother|BlackWidow|bmclient|Boston\ Project|BravoBrian\ SpiderEngine\ MarcoPolo|Bot\ mailto:craftbot@yahoo.com|Buddy|Bullseye|bumblebee|capture|CherryPicker|ChinaClaw|CICC|clipping|Collector|Copier|Crescent|Crescent\ Internet\ ToolPak|Custo|cyberalert|Deweb|diagem|Digger|Digimarc|DIIbot|DISCo|DISCo\ Pump|DISCoFinder|Download\ Demon|Download\ Wonder|Downloader|Drip|DSurf15a|DTS.Agent|EasyDL|eCatch|ecollector|efp@gmx\.net|Email\ Extractor|EirGrabber|email|EmailCollector|EmailSiphon|EmailWolf|Express\ WebPictures|ExtractorPro|EyeNetIE|FavOrg|fastlwspider|Favorites\ Sweeper|Fetch|FEZhead|FileHound|FlashGet\ WebWasher|FlickBot|fluffy|FrontPage|GalaxyBot|Generic|Getleft|GetRight|GetSmart|GetWeb!|GetWebPage|gigabaz|Girafabot|Go\!Zilla|Go!Zilla|Go-Ahead-Got-It|GornKer|gotit|Grabber|GrabNet|Grafula|Green\ Research|grub-client|Harvest|hhjhj@yahoo|hloader|HMView|HomePageSearch|http\ generic|HTTrack|httpdown|httrack|ia_archiver|IBM_Planetwide|Image\ Stripper|Image\ Sucker|imagefetch|IncyWincy|Indy*Library|Indy\ Library|informant|Ingelin|InterGET|Internet\ Ninja|InternetLinkagent|Internet\ Ninja|InternetSeer\.com|Iria|Irvine|JBH*agent|JetCar|JOC|JOC\ Web\ Spider|JustView|KWebGet|Lachesis|larbin|Leacher|LeechFTP|LexiBot|lftp|libwww|likse|Link|Link*Sleuth|LINKS\ ARoMATIZED|LinkWalker|LWP|lwp-trivial|Mag-Net|Magnet|Mac\ Finder|Mag-Net|Mass\ Downloader|MCspider|MJ12bot/v1\.0\.8|Memo|Microsoft.URL|MIDown\ tool|Mirror|Missigua\ Locator|Mister\ PiX|MMMtoCrawl\/UrlDispatcherLLL|^Mozilla$|Mozilla.*Indy|Mozilla.*NEWT|Mozilla*MSIECrawler|MS\ FrontPage*|MSFrontPage|MSIECrawler|MSProxy|multithreaddb|nationaldirectory|Navroad|NearSite|NetAnts|NetCarta|NetMechanic|netprospector|NetResearchServer|NetSpider|Net\ Vampire|NetZIP|NetZip\ Downloader|NetZippy|NEWT|NICErsPRO|Ninja|NPBot|Octopus|Offline\ Explorer|Offline\ Navigator|OpaL|Openfind|OpenTextSiteCrawler|OrangeBot|PageGrabber|Papa\ Foto|PackRat|pavuk|pcBrowser|PersonaPilot|Ping|PingALink|Pockey|Proxy|psbot|PSurf|psycheclone|puf|Pump|PushSite|QRVA|RealDownload|Reaper|Recorder|ReGet|replacer|RepoMonkey|Robozilla|Rover|RPT-HTTPClient|Rsync|Scooter|SearchExpress|searchhippo|searchterms\.it|Second\ Street\ Research|Seeker|Shai|Siphon|sitecheck|sitecheck.internetseer.com|SiteSnagger|SlySearch|SmartDownload|snagger|Snake|SpaceBison|Spegla|SpiderBot|sproose|SqWorm|Stripper|Sucker|SuperBot|SuperHTTP|Surfbot|SurfWalker|Szukacz|tAkeOut|tarspider|Teleport\ Pro|Templeton|TrueRobot|TV33_Mercator|UIowaCrawler|UtilMind|URLSpiderPro|URL_Spider_Pro|Vacuum|vagabondo|vayala|visibilitygap|VoidEYE|vspider|Web\ Downloader|w3mir|Web\ Data\ Extractor|Web\ Image\ Collector|Web\ Sucker|Wweb|WebAuto|WebBandit|web\.by\.mail|Webclipping|webcollage|webcollector|WebCopier|webcraft@bea|webdevil|webdownloader|Webdup|WebEMailExtrac|WebFetch|WebGo\ IS|WebHook|Webinator|WebLeacher|WEBMASTERS|WebMiner|WebMirror|webmole|WebReaper|WebSauger|Website|Website\ eXtractor|Website\ Quester|WebSnake|Webster|WebStripper|websucker|webvac|webwalk|webweasel|WebWhacker|WebZIP|Wget|Whacker|whizbang|WhosTalking|Widow|WISEbot|WWWOFFLE|x-Tractor|^Xaldon\ WebSpider|WUMPUS|Xenu|XGET|Zeus.*Webster|Zeus [NC]
RewriteRule ^.* - [F,L]

and

SetEnvIfNoCase User-Agent "8484 Boston Project v 1.0" bad_bot
SetEnvIfNoCase User-Agent "charlotte/" bad_bot
SetEnvIfNoCase User-Agent "curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5" bad_bot
SetEnvifNoCase User-Agent "ISC Systems iRc Search 2.1" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta\ Commons-HttpClient/" bad_bot
SetEnvIfNoCase User-Agent "Java 1.5 / IBM HTML Commons" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.1_01" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.1_04" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_01" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_02" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_03" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_04" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_05" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_07" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_08" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_09" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_10" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_12" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_13" bad_bot
SetEnvIfNoCase User-Agent "Java/1.4.2_16" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0-p3" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_01" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_02" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_03" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_04" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_05" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_06" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_07" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_08" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_09" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_10" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_11" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_12" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_13" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_14" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0-beta" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0-beta2" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0-beta" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0-beta2" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0-dp" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0-oem" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0-rc" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0_01" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0_01-ea" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0_02" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0_03" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0_04" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0_05" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0_06" bad_bot
SetEnvIfNoCase User-Agent "Java1.1.8" bad_bot
SetEnvIfNoCase User-Agent "Java1.2.2" bad_bot
SetEnvIfNoCase User-Agent "Java1.3.0" bad_bot
SetEnvIfNoCase User-Agent "Java1.3.1" bad_bot
SetEnvIfNoCase User-Agent "Java1.3.1_07" bad_bot
SetEnvIfNoCase User-Agent "Java1.3.1_18" bad_bot
SetEnvIfNoCase User-Agent "Java1.4.0" bad_bot
SetEnvIfNoCase User-Agent "Java1.4.0_01" bad_bot
SetEnvIfNoCase User-Agent "libwww-perl/" bad_bot
SetEnvIfNoCase User-Agent "^libcurl-agent/" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft\ URL\ Control.*$" bad_bot
SetEnvIfNoCase User-Agent "MJ12bot/v1.0.8" bad_bot
SetEnvIfNoCase User-Agent "^Missigua" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/4\.0\ .*Win\ 9x\ 4\.90.*$" bad_bot
SetEnvIfNoCase User-Agent "Nutch" bad_bot
SetEnvIfNoCase User-Agent "phpversion" bad_bot
SetEnvIfNoCase User-Agent "TencentTraveler" bad_bot
SetEnvIfNoCase User-Agent "^Web Downloader" bad_bot
<FilesMatch "(.*)">
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</FilesMatch>

In our robots.txt we are blocking also some bots, including the bots attempting articles theft:

User-agent: nutch
Disallow: / 

User-Agent: OmniExplorer_Bot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: Bitacle bot/1.1
Disallow: /

User-agent: Bitacle bot
Disallow: /

User-agent: Bitacle *
Disallow: /

User-agent: Bitacle*
Disallow: /

User-agent: Bitacle
Disallow: /

If you know any missing, please be kind a let us know so we can update.

The ones we could not block so far, we let Distributed Spam Harvester Tracking Network | Project Honey Pot to catch them.

Any further problems please?

Thanks for trying to support SEO Workers. Great promotion attempt!!!

[Burf.com]

Who else bans bots like this

Webnauts implemented this for me and its made quite a difference on traffic / bw etc

[Webnauts]

I just wanted to bring to your consideration that I just found some bots I did not manage to exclude so far.

I updated my files and the same time I updated my post above too.

Stay tuned.

[wige]

I like it. Going to have to try that method out on my site.

[Webnauts]

Originally Posted by wige

I like it. Going to have to try that method out on my site.

I updated this moment again. I added all Java bots I could find, and others too now.

I can't express with words how much bandwidth it saved us so far. And how much spam is reduced. Here you can follow up the most recent results: Statistics for seoworkers.com (2008-05)

Try it and you will be amazed too buddy.

[wige]

Here are a few others that I have blocked on mine:

Arachmo
BecomeBot
bot/1.0
Clushbot
Gigabot
kalooga
Kyluka crawl
MSR-ISRCCrawler
my-heritrix-crawler
NicheBot
noxtrumbot
Pingdom
POE-Component-Client-HTTP
Yeti
zermelo

[Webnauts]

Wige thanks for the tip in your PM.

I just replaced the all SetEnvIfNoCase for the Java User-Agents with the one you advised me

BrowserMatch "^Java/?[1-9_\.]*" bad_bot

and tested with this simulator several versions and looks like it works: Simulate any User Agent or Bot

[Webnauts]

Originally Posted by wige

Here are a few others that I have blocked on mine:

Arachmo
BecomeBot
bot/1.0
Clushbot
Gigabot
kalooga
Kyluka crawl
MSR-ISRCCrawler
my-heritrix-crawler
NicheBot
noxtrumbot
Pingdom
POE-Component-Client-HTTP
Yeti
zermelo

Wige thanks a lot for the heads up. I added those bots too, except of "Kyluka Crawl", since I could not get it right somehow. Maybe you can help there.

So here are the security updated settings in my .htaccess so far:

### Protect against DOS attacks by limiting file upload size ###
LimitRequestBody 10240000

### Prevent .htaccess and .htpasswd files from being viewed by web clients ###
<Files "^\.ht">
  Order allow,deny
  Deny from all
</Files>

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ADSARobot|ah-ha|almaden|aktuelles|Anarchie|amzn_assoc|Arachmo|ASPSeek|ASSORT|ATHENS|Atomz|attach|attache|autoemailspider|BackWeb|Bandit|BatchFTP|bdfetch|BecomeBot|big.brother|BlackWidow|bmclient|Boston\ Project|bot/1.0|BravoBrian\ SpiderEngine\ MarcoPolo|Bot\ mailto:craftbot@yahoo.com|Buddy|Bullseye|bumblebee|capture|CherryPicker|ChinaClaw|CICC|clipping|Clushbot|Collector|Copier|Crescent|Crescent\ Internet\ ToolPak|Custo|cyberalert|Deweb|diagem|Digger|Digimarc|DIIbot|DISCo|DISCo\ Pump|DISCoFinder|Download\ Demon|Download\ Wonder|Downloader|Drip|DSurf15a|DTS.Agent|EasyDL|eCatch|ecollector|efp@gmx\.net|Email\ Extractor|EirGrabber|email|EmailCollector|EmailSiphon|EmailWolf|Express\ WebPictures|ExtractorPro|EyeNetIE|FavOrg|fastlwspider|Favorites\ Sweeper|Fetch|FEZhead|FileHound|FlashGet\ WebWasher|FlickBot|fluffy|FrontPage|GalaxyBot|Generic|Getleft|GetRight|GetSmart|GetWeb!|GetWebPage|gigabaz|Gigabot|Girafabot|Go\!Zilla|Go!Zilla|Go-Ahead-Got-It|GornKer|gotit|Grabber|GrabNet|Grafula|Green\ Research|grub-client|Harvest|hhjhj@yahoo|hloader|HMView|HomePageSearch|http\ generic|HTTrack|httpdown|httrack|ia_archiver|IBM_Planetwide|Image\ Stripper|Image\ Sucker|imagefetch|IncyWincy|Indy*Library|Indy\ Library|informant|Ingelin|InterGET|Internet\ Ninja|InternetLinkagent|Internet\ Ninja|InternetSeer\.com|Iria|Irvine|JBH*agent|JetCar|JOC|JOC\ Web\ Spider|JustView|kalooga|KWebGet|Lachesis|larbin|Leacher|LeechFTP|LexiBot|lftp|libwww|likse|Link|Link*Sleuth|LINKS\ ARoMATIZED|LinkWalker|LWP|lwp-trivial|Mag-Net|Magnet|Mac\ Finder|Mag-Net|Mass\ Downloader|MCspider|MJ12bot/v1\.0\.8|Memo|Microsoft.URL|MIDown\ tool|Mirror|Missigua\ Locator|Mister\ PiX|MMMtoCrawl\/UrlDispatcherLLL|^Mozilla$|Mozilla.*Indy|Mozilla.*NEWT|Mozilla*MSIECrawler|MS\ FrontPage*|MSFrontPage|MSIECrawler|MSProxy|MSR-ISRCCrawler|multithreaddb|my-heritrix-crawler|nationaldirectory|Navroad|NearSite|NetAnts|NetCarta|NetMechanic|netprospector|NetResearchServer|NetSpider|Net\ Vampire|NetZIP|NetZip\ Downloader|NetZippy|NEWT|NICErsPRO|Ninja|NPBot|NicheBot|noxtrumbot|Octopus|Offline\ Explorer|Offline\ Navigator|OpaL|Openfind|OpenTextSiteCrawler|OrangeBot|PageGrabber|Papa\ Foto|PackRat|pavuk|pcBrowser|PersonaPilot|Ping|PingALink|Pingdom|Pockey|POE-Component-Client-HTTP|Proxy|psbot|PSurf|psycheclone|puf|Pump|PushSite|QRVA|RealDownload|Reaper|Recorder|ReGet|replacer|RepoMonkey|Robozilla|Rover|RPT-HTTPClient|Rsync|Scooter|SearchExpress|searchhippo|searchterms\.it|Second\ Street\ Research|Seeker|Shai|Siphon|sitecheck|sitecheck.internetseer.com|SiteSnagger|SlySearch|SmartDownload|snagger|Snake|SpaceBison|Spegla|SpiderBot|sproose|SqWorm|Stripper|Sucker|SuperBot|SuperHTTP|Surfbot|SurfWalker|Szukacz|tAkeOut|tarspider|Teleport\ Pro|Templeton|TrueRobot|TV33_Mercator|UIowaCrawler|UtilMind|URLSpiderPro|URL_Spider_Pro|Vacuum|vagabondo|vayala|visibilitygap|VoidEYE|vspider|Web\ Downloader|w3mir|Web\ Data\ Extractor|Web\ Image\ Collector|Web\ Sucker|Wweb|WebAuto|WebBandit|web\.by\.mail|Webclipping|webcollage|webcollector|WebCopier|webcraft@bea|webdevil|webdownloader|Webdup|WebEMailExtrac|WebFetch|WebGo\ IS|WebHook|Webinator|WebLeacher|WEBMASTERS|WebMiner|WebMirror|webmole|WebReaper|WebSauger|Website|Website\ eXtractor|Website\ Quester|WebSnake|Webster|WebStripper|websucker|webvac|webwalk|webweasel|WebWhacker|WebZIP|Wget|Whacker|whizbang|WhosTalking|Widow|WISEbot|WWWOFFLE|x-Tractor|^Xaldon\ WebSpider|WUMPUS|Xenu|XGET|Yeti|zermelo|Zeus.*Webster|Zeus [NC]
RewriteRule ^.* - [F,L]

### Deny Fake Bots ###
BrowserMatch "^Java/?[1-9_\.]*" bad_bot
SetEnvIfNoCase User-Agent "8484 Boston Project v 1.0" bad_bot
SetEnvIfNoCase User-Agent "charlotte/" bad_bot
SetEnvIfNoCase User-Agent "curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5" bad_bot
SetEnvifNoCase User-Agent "ISC Systems iRc Search 2.1" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta\ Commons-HttpClient/" bad_bot
SetEnvIfNoCase User-Agent "libwww-perl/" bad_bot
SetEnvIfNoCase User-Agent "^libcurl-agent/" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft\ URL\ Control.*$" bad_bot
SetEnvIfNoCase User-Agent "MJ12bot/v1.0.8" bad_bot
SetEnvIfNoCase User-Agent "^Missigua" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/4\.0\ .*Win\ 9x\ 4\.90.*$" bad_bot
SetEnvIfNoCase User-Agent "Nutch" bad_bot
SetEnvIfNoCase User-Agent "phpversion" bad_bot
SetEnvIfNoCase User-Agent "TencentTraveler" bad_bot
SetEnvIfNoCase User-Agent "^Web Downloader" bad_bot
<FilesMatch "(.*)">
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</FilesMatch>

Any further ideas to optimize (shrink/compress) the above?

I was thinking also to block proxy servers, but I am not sure if that would have been a good idea.
What do you think about this?

### Block proxy servers from site access ###
RewriteCond %{HTTP:VIA}                 !^$ [OR]
RewriteCond %{HTTP:FORWARDED}           !^$ [OR]
RewriteCond %{HTTP:USERAGENT_VIA}       !^$ [OR]
RewriteCond %{HTTP:X_FORWARDED_FOR}     !^$ [OR]
RewriteCond %{HTTP:PROXY_CONNECTION}    !^$ [OR]
RewriteCond %{HTTP:XPROXY_CONNECTION}   !^$ [OR]
RewriteCond %{HTTP:HTTP_PC_REMOTE_ADDR} !^$ [OR]
RewriteCond %{HTTP:HTTP_CLIENT_IP}      !^$
RewriteRule ^(.*)$ - [F]

[kgun]

Originally Posted by Burf.com

Who else bans bots like this

Webnauts implemented this for me and its made quite a difference on traffic / bw etc

Try http://www.forumnorway.com/

Ideally it blocks all non-Norwegian Ip's.

Not a single attack in months.

[Webnauts]

Originally Posted by kgun

Try http://www.forumnorway.com/

Ideally it blocks all non-Norwegian Ip's

That was a good one.

Is there also an ideal solution to block all India IPs?

[Webnauts]

Originally Posted by kgun

Try http://www.forumnorway.com/

Ideally it blocks all non-Norwegian Ip's.

Not a single attack in months.

Are you claiming that your forums cannot be viewed through a Norvegian proxy?

[Webnauts]

Originally Posted by subhzash

Hi All,

When i want to see the internal links of SEO Workers Search Engine Optimization Consulting Company through dead-links.com and also with XENU
It is giving following error

Dead-links.com shows this error

SEO Workers Search Engine Optimization Consulting Company 403 Forbidden

XENU shows that

No such host



Regards
Subhzash

Subhzash, where did you go? Are you still sure that something was wrong?

[kgun]

Originally Posted by Webnauts

Are you claiming that your forums cannot be viewed through a Norvegian proxy?

May be. John try.

[kgun]

Originally Posted by Webnauts

That was a good one.

Is there also an ideal solution to block all India IPs?

Done in a few minutes block a country, block by ip address, deny access by country

Additional information here: DigitalStart.net: The starting point for English speaking surfers and webmasters

[Webnauts]

Originally Posted by kgun

May be. John try.

I do not need to try. If I come through a Norwegian proxy server to your forums, I will have access.

[Webnauts]

Originally Posted by kgun

Done in a few minutes block a country, block by ip address, deny access by country

Well I am aware of that solution already. But my .htaccess is already huge.

Originally Posted by kgun

Additional information here: DigitalStart.net: The starting point for English speaking surfers and webmasters

Congrats brother. I like the idea!!! I wish I had an IBL on the page.

[wige]

Well, already I have seen a distinct drop in bandwidth usage today. I am using the BrowserMatch "whatever" bad_bot method, because I can set this in the server's configuration and apply it to all of the virtual hosts, which I could not get working with the RewriteRule for some reason. Probably had a conflict somewhere.

[Webnauts]

Originally Posted by wige

Well, already I have seen a distinct drop in bandwidth usage today. I am using the BrowserMatch "whatever" bad_bot method, because I can set this in the server's configuration and apply it to all of the virtual hosts, which I could not get working with the RewriteRule for some reason. Probably had a conflict somewhere.

Great to hear positive news!

By the way I would like to ask you probably use Skype or IM? And do you speak Perl?

[subhzash]

Originally Posted by Webnauts

Subhzash, where did you go? Are you still sure that something was wrong?

Not really. I dont think still some thing is wrong in your website. But my main intension is to prove my self by finding bugs in your site. Some times your words really worst and those hurt me.

Anyway not a problem...But still i do some efforts to find bugs in your site.



Regards
Subhzash

[full house]

do you detect spammers as easy as that?

[wige]

Originally Posted by Webnauts

Great to hear positive news!

By the way I would like to ask you probably use Skype or IM? And do you speak Perl?

I do use IM, just added them to my profile, although I am on MSN a lot more than Yahoo. As for Perl, I used to build my web sites entirely in Perl (Yes, I did once say "PHP? Bah! Merely a fad - real devs use Perl!" - sorta) but its been so long since I used it I have forgotten most of it.

Originally Posted by subhzash

Not really. I dont think still some thing is wrong in your website. But my main intension is to prove my self by finding bugs in your site. Some times your words really worst and those hurt me.

Anyway not a problem...But still i do some efforts to find bugs in your site.



Regards
Subhzash

There are a few. Need to be a hacker to spot them though.

Originally Posted by full house

do you detect spammers as easy as that?

If you mean spambots, I use a database to log visits so I can track how popular different products are, and to allow certain customization of the site for visitors. I tweaked the logging system to spot user agents that are only on a few IP addresses, or that visit on unusual rates, and from that I can spot both new browsers (for example, an upswing in traffic from mobile browsers) and spam bots for blocking. If you are referring to forum and form spam, a careful approach to security and input validation, combined with a system that tracks unexpected user input and blocks the source, can go a long way toward eliminating the problem.

[Webnauts]

Originally Posted by subhzash

Not really. I dont think still some thing is wrong in your website. But my main intension is to prove my self by finding bugs in your site. Some times your words really worst and those hurt me.

Anyway not a problem...But still i do some efforts to find bugs in your site.



Regards
Subhzash

Hey sorry man. Now I understand entirely your intention. I honestly thank you for your kind efforts to support me perfect my site.

Can we be friends now?

[Webnauts]

Originally Posted by wige

Here are a few others that I have blocked on mine:

Arachmo
BecomeBot
bot/1.0
Clushbot
Gigabot
kalooga
Kyluka crawl
MSR-ISRCCrawler
my-heritrix-crawler
NicheBot
noxtrumbot
Pingdom
POE-Component-Client-HTTP
Yeti
zermelo

Wige are you sure GigaBot is a spambot? I made a research and I found out that it is the bot of the search engine Gigablast.

[kgun]

Originally Posted by Webnauts

Wige are you sure GigaBot is a spambot? I made a research and I found out that it is the bot of the search engine Gigablast.

If you are a member of WMW, you may have noticed that some members think that nearly every bot is bad.

Even the old well established ex Norwegian SE FAST - Enterprise Search now owned by Microsoft was blocked by some members.

[Webnauts]

Originally Posted by kgun

Even the old well established ex Norwegian SE FAST - Enterprise Search now owned by Microsoft was blocked by some members.

WHAAAT????

Off-topic: My online support is active at the moment.

[subhzash]

Originally Posted by Webnauts

Hey sorry man. Now I understand entirely your intention. I honestly thank you for your kind efforts to support me perfect my site.

Can we be friends now?

Sure. Thanks webnauts. I feel happy to have friend like to you. I would like to add your yahoo and gmail ids to add my friends list. Please send me


Thanks a lot


Regards
Subhzash

[subhzash]

I hope dead-links.com is not bad bot. I use to list out URLs of the website which have less than 250 pages.

Why do you stop dead-links.com to crawl the website??

[Webnauts]

Originally Posted by subhzash

I hope dead-links.com is not bad bot. I use to list out URLs of the website which have less than 250 pages.

Why do you stop dead-links.com to crawl the website??

We check if we have dead links ourselves. Every time we do changes on our site, we turn off temporary the rules that disallow our tools to check, and when things are fine we block everything again.
What is wrong with that?

By the way a great news! Where did the bad bots go?
Statistics for seoworkers.com (2008-06)

[subhzash]

Originally Posted by Webnauts

We check if we have dead links ourselves. Every time we do changes on our site, we turn off temporary the rules that disallow our tools to check, and when things are fine we block everything again.
What is wrong with that?

By the way a great news! Where did the bad bots go?
Statistics for seoworkers.com (2008-06)

Nothing wrong. Actually i had doubt that how do you check internal links if some internal site checking bots are blocked. Any way my doubt clarified.

[wige]

Originally Posted by Webnauts

Wige are you sure GigaBot is a spambot? I made a research and I found out that it is the bot of the search engine Gigablast.

Ah. I will remove it from my block list.

Originally Posted by kgun

Even the old well established ex Norwegian SE FAST - Enterprise Search now owned by Microsoft was blocked by some members.

That one too.

Originally Posted by kgun

If you are a member of WMW, you may have noticed that some members think that nearly every bot is bad.

Yeah, I noticed some even recommend blocking Google's Mobile Transcoder, a human operated (thus not a bot) service that converts standard web sites for mobile users, because it doesn't obey robots.txt.

[kgun]

Originally Posted by wige

That one too.

I don't kow if you use this Help : when you search that forum.

I think you find it in a thread with the following KW's

ban a country using htaccess

Here is the thread: WebmasterWorld Login

"i want to banned certain ip adresses all coming from:
UserAgent: FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
IP address: 66.77.73.151

i get way to many visits from this spiderbot"

My bolding.

Note the answer (last post) from heini:

"Just a quick and offtopic addition: you are about to block the spider of one of the few big worldwide search engines".

[Webnauts]

Originally Posted by kgun

Note the answer (last post) from heini:

"Just a quick and offtopic addition: you are about to block the spider of one of the few big worldwide search engines".

Do you probably mean one of the best?

[kgun]

What do you mean by best?

I started a new thread about this important topic here:

Google the best search engine?

But I noted an interesting difference while using this search form on the big three and the KW's

link selling

Google 10 200 hits

Yahoo 9 700 hits

MSN 20 200 hits

Is that a result of

• Indexing
• Reporting
• New MSN search algorithms

Took a cup of coffee outside in the beautiful may weather and got back to the computer and tried these searches on Google:

link selling site:WebmasterWorld News and Discussion for the Web Professional (www-version) 10 100 hits

link selling site:webmasterworld.com 88 500

from webmasterworld.com OR webmasterworld.com for link selling.

So the reason was as I thought how WMW has programmed that form.

On MSN, the www and non-www version produces approximately the same number of hits.

link selling site:webmasterworld.com 20 000 hits

link selling site:WebmasterWorld News and Discussion for the Web Professional (www-version) 20 300 hits