minstrel
12-28-2003, 07:27 PM
From the Alexa website (http://pages.alexa.com/help/webmasters/index.html?p=Corp_W_t_40_L1#prevent):
Alexa Information for Webmasters
You may be wondering about "ia_archiver" and be curious about why it is visiting your site, or you may want to invite the robot to crawl your site. To block ia_archiver from crawling your site, please read below.
Additional information regarding our privacy policy, web crawling philosophy, and technology can be found on the following pages Privacy Policy and Technology. If you wish to change the contact information for your site, please visit our contact information editor. If you would like to suggest some Related Links, please visit our related link suggestion page.
The Alexa crawler (robot), which identifies itself as ia_archiver in the HTTP "User-agent" header field, uses a web-wide crawl strategy. Basically, it starts with a list of known URLs from across the entire Internet, then it fetches all local links found as it goes. There are several advantages to this approach, most importantly that it creates the least possible disruption to the sites being crawled.
We will not index anything you would like to remain private. All you have to do is tell us. How? By using the Standard for Robot Exclusion (SRE).
The SRE was developed by Martijn Koster at Webcrawler to allow content providers to control how robots behave on their sites. All of the major Web-crawling groups, such as AltaVista, Inktomi, and Google, respect this standard. Alexa Internet strictly adheres to the standard:
Whenever ia_archiver lands on the top level of a Web site, it looks for a file called "robots.txt". Robots.txt is a file website administrators can place at the top level of a site to direct the behavior of web crawling robots.
The Alexa crawler will always pick up a copy of the robots.txt file prior to its crawl of the Web. If you change your robots.txt file while we are crawling your site, please let us know so that we can instruct the crawler to retrieve the updated instructions contained in the robots.txt file.
After retrieving any HTML file, we check for the presence of the NOINDEX, NOARCHIVE, and NOFOLLOW tags in the "<head>" element of the document. If we find a NOINDEX or NOARCHIVE tag, we throw away the copy. If there is a NOFOLLOW tag, the robot will not follow any links found on that page. This allows users to control access to their own data, without needing their site administrators to update "robots.txt".
To exclude all robots, the robots.txt file should look like this:
User-agent: *
Disallow: /
To exclude just one directory (and its subdirectories), say, the /images/ directory, the file should look like this:
User-agent: *
Disallow: /images/
Web site administrators can allow or disallow specific robots from visiting part or all of their site. Alexa's crawler identifies itself as ia_archiver, and so to allow ia_archiver to visit (while preventing all others), your robots.txt file should look like this:
User-agent: ia_archiver
Disallow:
To prevent ia_archiver from visiting (while allowing all others), your robots.txt file should look like this:
User-agent: ia_archiver
Disallow: /
For more information regarding robots, crawling, and robots.txt visit the Web Robots Pages at www.robotstxt.org, an excellent source for the latest information on the Standard for Robots Exclusion.
Fill out the form below to be crawled by Alexa.
There are a few reasons that Alexa may not have visited your site. Perhaps your site is new or we haven't discovered any links on the web that lead to your site. Or perhaps we haven't had any Alexa users visit your site. It is also possible that your web site administrator has disallowed crawlers from visiting your site - please read the information about robots.txt that we have provided above.
In any event, simply by visiting your site with the Alexa Toolbar open, Alexa will learn of your site and add it to our list of sites to visit, thus ensuring your inclusion in the Alexa service and in the Alexa archive.
Alexa Information for Webmasters
You may be wondering about "ia_archiver" and be curious about why it is visiting your site, or you may want to invite the robot to crawl your site. To block ia_archiver from crawling your site, please read below.
Additional information regarding our privacy policy, web crawling philosophy, and technology can be found on the following pages Privacy Policy and Technology. If you wish to change the contact information for your site, please visit our contact information editor. If you would like to suggest some Related Links, please visit our related link suggestion page.
The Alexa crawler (robot), which identifies itself as ia_archiver in the HTTP "User-agent" header field, uses a web-wide crawl strategy. Basically, it starts with a list of known URLs from across the entire Internet, then it fetches all local links found as it goes. There are several advantages to this approach, most importantly that it creates the least possible disruption to the sites being crawled.
We will not index anything you would like to remain private. All you have to do is tell us. How? By using the Standard for Robot Exclusion (SRE).
The SRE was developed by Martijn Koster at Webcrawler to allow content providers to control how robots behave on their sites. All of the major Web-crawling groups, such as AltaVista, Inktomi, and Google, respect this standard. Alexa Internet strictly adheres to the standard:
Whenever ia_archiver lands on the top level of a Web site, it looks for a file called "robots.txt". Robots.txt is a file website administrators can place at the top level of a site to direct the behavior of web crawling robots.
The Alexa crawler will always pick up a copy of the robots.txt file prior to its crawl of the Web. If you change your robots.txt file while we are crawling your site, please let us know so that we can instruct the crawler to retrieve the updated instructions contained in the robots.txt file.
After retrieving any HTML file, we check for the presence of the NOINDEX, NOARCHIVE, and NOFOLLOW tags in the "<head>" element of the document. If we find a NOINDEX or NOARCHIVE tag, we throw away the copy. If there is a NOFOLLOW tag, the robot will not follow any links found on that page. This allows users to control access to their own data, without needing their site administrators to update "robots.txt".
To exclude all robots, the robots.txt file should look like this:
User-agent: *
Disallow: /
To exclude just one directory (and its subdirectories), say, the /images/ directory, the file should look like this:
User-agent: *
Disallow: /images/
Web site administrators can allow or disallow specific robots from visiting part or all of their site. Alexa's crawler identifies itself as ia_archiver, and so to allow ia_archiver to visit (while preventing all others), your robots.txt file should look like this:
User-agent: ia_archiver
Disallow:
To prevent ia_archiver from visiting (while allowing all others), your robots.txt file should look like this:
User-agent: ia_archiver
Disallow: /
For more information regarding robots, crawling, and robots.txt visit the Web Robots Pages at www.robotstxt.org, an excellent source for the latest information on the Standard for Robots Exclusion.
Fill out the form below to be crawled by Alexa.
There are a few reasons that Alexa may not have visited your site. Perhaps your site is new or we haven't discovered any links on the web that lead to your site. Or perhaps we haven't had any Alexa users visit your site. It is also possible that your web site administrator has disallowed crawlers from visiting your site - please read the information about robots.txt that we have provided above.
In any event, simply by visiting your site with the Alexa Toolbar open, Alexa will learn of your site and add it to our list of sites to visit, thus ensuring your inclusion in the Alexa service and in the Alexa archive.