iEntry 10th Anniversary Forum Rules Search
WebProWorld
Register FAQ Calendar Mark Forums Read
Search Engine Optimization Forum SEO is much easier with help from peers and experts! The WebProWorld SEO forum is for the discussion and exploration of various search engine optimization topics. Any non (engine) specific SEO or SEM topics should go here.

Share Thread: & Tags

Share Thread:

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 08-06-2004, 09:18 AM
WebProWorld New Member
 
Join Date: Aug 2004
Location: International
Posts: 5
NewMediaDiva RepRank 0
Default How do you know when you've been crawled?

I feel really stupid for asking but...how exactly does everyone seem to know (down to the day!) when they've been crawled? And how do you know who it is that has crawled?

Is there an easy way to check this, other than digging into logs and webstats?
Reply With Quote
  #2 (permalink)  
Old 08-06-2004, 09:53 AM
pne's Avatar
pne pne is offline
WebProWorld Pro
 
Join Date: May 2004
Location: UK
Posts: 292
pne RepRank 0
Default

Diva,

What kind of web logs are you using/accessing? Major bots are

Googlebot

Yahoo!Slurp

MSN bot

They should be clearly identifiable from your web logs.

pne
__________________
<a href="http://www.sochoose.com/" target="_blank">Employee Assistance Programme
<a href="http://www.sochoose.com/employee_wellness_programme.php"/target="_blank">Employee Wellness Programme
Reply With Quote
  #3 (permalink)  
Old 08-06-2004, 01:12 PM
WebProWorld Pro
 
Join Date: Mar 2004
Location: UK
Posts: 203
buddhu RepRank 0
Default

Hi Diva,

It might be helpful to get a list of the IP addresses used by various crawlers.

Check out http://www.searchengineworld.com/spiders/spider_ips.htm
Reply With Quote
  #4 (permalink)  
Old 08-06-2004, 03:55 PM
WebProWorld Member
 
Join Date: Apr 2004
Location: San Diego
Posts: 43
AITechnology RepRank 0
Default

I had trouble with my host to check my log file so I had to make my own log file. I got it from a website and have NO memory of where I got it from so if the owner of this wants to take credit, please do so.

Put this text in your index file and rename it as a index.php Place this just below the header. I included the </head> as a guide.

</head>
<?php
$file = fopen("log.html", "a");

$time = date("H:i dS F");
fwrite($file, "Time: $time
" );

if( $REMOTE_ADDR != null)
{
fwrite($file,"Ip address: $REMOTE_ADDR
");
}

if( $HTTP_REFERER != null)
{
fwrite($file,"Referer: $HTTP_REFERER
");
}

fwrite($file,"Browser: $HTTP_USER_AGENT<hr/>");

fclose($file)

?>

Next...you want to create a new file and call it log.php and put this code into it...

<?php
$file = fopen("log.html", "a");

$time = date("H:i dS F");
fwrite($file, "Time: $time
" );

if( $REMOTE_ADDR != null)
{
fwrite($file,"Ip address: $REMOTE_ADDR
");
}

if( $HTTP_REFERER != null)
{
fwrite($file,"Referer: $HTTP_REFERER
");
}

fwrite($file,"Browser: $HTTP_USER_AGENT<hr/>");

fclose($file)

?>

Finally, create an empty file called log.html
Very important to set the permissions on the log file so that it can be written too. The recommendation was to give it a 7 7 7 permission, but I changed mine and only look at the file via Dreamweaver so that jo schmo cant just look up this info.

Also, remember to empty it out every so often otherwise it can get quite big.


Hope this helps.
Reply With Quote
Reply

  WebProWorld > Search Engines > Search Engine Optimization Forum

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 02:22 AM.



Search Engine Optimization by vBSEO 3.3.0