Google introduced a new META tag that allows you to tell them when a page should be removed from the main Google web search results called unavailable_after tag.


This one follows a similar syntax to other META tags. For example, to specify that an HTML page should be removed from the search results after 8pm Eastern Standard Time on 19th August 2007, simply add the following tag to the first section of your page:

<META NAME="GOOGLEBOT" CONTENT="unavailable_after: 19-Aug-2007 08:00:00 EST">

But how could you tell the robots that the site is unavailable between certain hours of the day?


An Extended Standard for Robot Exclusion has been proposed, which adds several new directives, such as Visit-time. For example:

User-agent: *
Disallow: /downloads/
Visit-time: 0600-0845 # only visit between 6:00 AM and 8:45 AM UT (GMT)

What do you think about that?

Do you think this would work too?

User-agent: *
Disallow: /downloads/
Visit-time: 19-Aug-2007 0600-0845 # only visit on the 19th of August 2007 between 6:00 AM and 8:45 AM UT (GMT)

You contribution would be very appreciated.