How do you know when you've been crawled?

[NewMediaDiva]

I feel really stupid for asking but...how exactly does everyone seem to know (down to the day!) when they've been crawled? And how do you know who it is that has crawled?

Is there an easy way to check this, other than digging into logs and webstats?

[pne]

Diva,

What kind of web logs are you using/accessing? Major bots are

Googlebot

Yahoo!Slurp

MSN bot

They should be clearly identifiable from your web logs.

pne

[buddhu]

Hi Diva,

It might be helpful to get a list of the IP addresses used by various crawlers.

Check out http://www.searchengineworld.com/spiders/spider_ips.htm

[AITechnology]

I had trouble with my host to check my log file so I had to make my own log file. I got it from a website and have NO memory of where I got it from so if the owner of this wants to take credit, please do so.

Put this text in your index file and rename it as a index.php Place this just below the header. I included the </head> as a guide.

</head>
<?php
$file = fopen("log.html", "a");

$time = date("H:i dS F");
fwrite($file, "Time: $time
" );

if( $REMOTE_ADDR != null)
{
fwrite($file,"Ip address: $REMOTE_ADDR
");
}

if( $HTTP_REFERER != null)
{
fwrite($file,"Referer: $HTTP_REFERER
");
}

fwrite($file,"Browser: $HTTP_USER_AGENT<hr/>");

fclose($file)

?>

Next...you want to create a new file and call it log.php and put this code into it...

<?php
$file = fopen("log.html", "a");

$time = date("H:i dS F");
fwrite($file, "Time: $time
" );

if( $REMOTE_ADDR != null)
{
fwrite($file,"Ip address: $REMOTE_ADDR
");
}

if( $HTTP_REFERER != null)
{
fwrite($file,"Referer: $HTTP_REFERER
");
}

fwrite($file,"Browser: $HTTP_USER_AGENT<hr/>");

fclose($file)

?>

Finally, create an empty file called log.html
Very important to set the permissions on the log file so that it can be written too. The recommendation was to give it a 7 7 7 permission, but I changed mine and only look at the file via Dreamweaver so that jo schmo cant just look up this info.

Also, remember to empty it out every so often otherwise it can get quite big.

Hope this helps.