Google has introduced some new functionality to its Site Search product, which allows customers to search within your own site. The feature that everyone is talking about is the ability to add pages on-demand to be indexed by site search, so customers have access to these new pages as soon as they’re ready.
Blogging On Google Alerts Not Crawling For Blogs
Many people get Google news alerts. They’re pretty simple. You pick your topic, say the timely subject of hurricanes. When hurricane stories appear, you get an email. You can pick the frequency upon which they appear and whether or not the alert scoura just the news or the entire web.
Crawling Webs With Search Engine Spiders
When it comes to discussing search engine spiders, this is no time to succumb to feelings of terror. These are not the eight legged variety.
Microsoft Crawling Google Results For New Search Engine?
I was questioned today by a developer who was watching a particular IP address scan his site. The IP was 65.54.188.86 and is registered to Microsoft Corp. located at One Microsoft Way, Redmond, Washington 98052. This visitor was not sending the normal header information associated with a crawler to the web server such as an http robot name or identifying info or even a browser name.
Disabling Google and Other Search Engines From Crawling a Site
Reader question: I have a online database of horror movies, and I have a good Google rank. In my traffic logs I noted the last month a really growing of the bandwidth: one of the most important browsers of the server logs is Googlebot, so this traffic was generated for the spidering engine of Google. I have the 20 Gb bandwidth limit and I don’t want to pay for excess, so I disable Google into my Web site. My question is:
How Deep into Your Site Are the Spiders Crawling?
Have you ever wondered how far down into your site the search engine spiders are going? Are the pay inclusion spiders coming at regular intervals like they’re supposed to?