Andy Beard has hit upon a compromise between removing text links from his site and being summarily punished by Google for selling them.
Yahoo Search Supports X-Robots-Tags
Webmaster life got a little easier as Yahoo Search, in the midst of some updates to the engine, added page level exclusion tags to what it can handle.
Publishers Push ACAP As Robots.txt Improvement
The Automated Content Access Protocol (ACAP) debuted today as a set of improvements to deficiencies seen in the robots.txt protocol currently observed by search crawlers.
Google Teaches Robots Tool About Sitemaps
The robots.txt analysis tool at Google Webmaster Central received some much-needed updating, and should be more effective for webmasters today.
Google Flexes Robots Exclusion Protocol
Two new features added to the protocol will help webmasters govern when an item should stop showing up in Google’s web search, as well as providing some control over the indexing of other data types.
Yahoo Gets Sectional With Robots.txt
Yahoo’s spiders will obey a new tag, called robots-nocontent, that will allow webmasters to discretely define content on a page they do not want to have indexed.
Get Your Sitemap Discovered Automatically with your Robots.txt File
At Search Engine Strategies New York it was announced that you can now have your sitemap automatically discovered by configuring it in your Robots.txt file. It is simple and easy to do, you’ll just need to know the URL or web address of your sitemap.
Robots for the Rest of Us
The robots.txt file.
Nothing can be more confusing to a website owner as the robots.txt file. Born out of technology in the programming world, the robot.txt file is nothing more than a server command for search engines. Unfortunately, while search engines understand the file, humans have a difficult time understanding machine language.
Google On Robots Exclusion Protocol
A post on official Google blog informs about Robots Exclusion Protocol. Sometime back we informed you about a previous post on Robots.txt file.
Using Robots.txt To Prevent Search Indexing
Sometimes there are parts of your website you don’t want accessed by the search engines – for any number of reasons, like sensitive private data, articles that require subscriptions – whatever.