Beginning Level PHP Security Logging

[Tech Manager]

As an IT Manager a large amount of my daily time is spent on examining server logs looking for security issues, examining trends and tracking various forms and types of other data. Logs are a very important component of managing all types of servers. They are also an excellent tool to help you gather traffic data on your website(s).

This topic is not meant to be an in depth study of server and web logs, but merely an informational tool for those of you who may not have access to your web server logs or those who wish to create some quick statistical data related to your site visitors and data your visitors may input into forms. I am limiting this topic to gathering, storing and displaying a very small amount of data just to give you some ideas on how you might use an SSI language like PHP to dynamically develop some logging tools for your own personal use. If this topic garners enough interest I will expand it into some more advanced logging procedures including security responses.

In the early stages of the internet explosion many websites displayed a small hit counter at the bottom of the homepage. These displays were usually unreliable and often used to imply high volumes of traffic on a low volume website. As a webmaster, it will help you to get a good idea of the traffic (including origin & volume) coming to your website. For our scenario we will assume you are using PHP and are looking to gather data on three important visitor variables: IP Address, Date & Time, and a Search Term the visitor may enter into a search field on a web form located on your website. We will store this data in a text file located in a protected directory we will call site_logs.

Here are some simple steps for you to follow:

On your web server create a writeable directory called site_logs (located in your root directory).

In this directory add an .htaccess file that looks something like this:

 
<Limit GET HEAD POST>
Order Allow,Deny
</LIMIT>

The site_logs directory will be used to store the specific log file we create and the .htaccess file will be used to prevent prying eyes from viewing the contents over the web. The .htaccess file above is just a very simple example. Please feel free to modify this file to fit your needs.

Create a file called gather_data.php (or anything else you want to call this file).

The gather_data.php file will collect specific data and write it to a file in your site_logs directory.

The gather_data.php file will look like this:

 
<?php
 
$handle = fopen("site_logs/web_log.txt", "a+");
 
                                                // Set File Permissions to 644
            chmod("site_logs/web_log.txt", 0644);
 
                                                // Get date & time
            $date_time = date("l, F j, Y g:i:s", time());
                                                // Get GMT
                         $gmt = date("O", time());
            $date_time_gmt = "$date_time GMT$gmt";
 
            $fp = fopen("site_logs/web_log.txt", "a");
 
            $s_remote= $date_time_gmt . " " . 'REMOTE_ADDR: ' . " " . $_SERVER['REMOTE_ADDR'] . "\n";
            fwrite($fp, $s_remote);
 
            $s_remote= 'Search Term: ' . $_POST['search_term'] . "\n";
            fwrite($fp, $s_remote);
 
fclose($handle);
 
?>

If you are a newbie, don't be overwhelmed by the above code. I'll provide you with a brief overview:

Within the PHP tags (<?php ?>) the first thing we want to do is to create a filehandler for the text file we will be using to store the data. We want this file to be appendable (we want it to continue to add data instead of overwriting existing data).

$handle = fopen("site_logs/web_log.txt", "a+");

We create a file handling variable called $handle and use it to open (fopen function) the site_logs/ directory and the web_log.txt for appending (a+). If the web_log.txt file does not exist it will be created the first time this script is successfully used.

The next thing we do is set the file permissions (assuming a Linux server) to 644, Using the chmod function.

// Set File Permissions to 644 (this is just a coding comment)
chmod("site_logs/web_log.txt", 0644);

Now we will get the date/time & GMT for the webserver using three variables (We are using three variables just so you can see everything as it happens).

// Get date & time
$date_time = date("l, F j, Y g:i:s", time());

Using the date(), time() functions we get the date and time and format it so it will appear something like this:
Monday, January 21, 2008 3:34:19

Then, using the date(), time() functions we get the Greenwich Mean Time (GMT) for the local webserver (yes, we could've combined date/time/GMT into one simple function).

// Get GMT
$gmt = date("O", time());

This produces an output, depending on your server location/timezone, something like this:

-0600

Then we combine these results into one variable: $date_time_gmt = "$date_time GMT$gmt";

and prepare to append the results into our web_log.txt file:

$fp = fopen("site_logs/web_log.txt", "a");

Next, we append the data, including the IP address (using the environmental variable: $_SERVER['REMOTE_ADDR']) of the site visitor to the log file using the fwrite() function:

$s_remote= $date_time_gmt . " " . 'REMOTE_ADDR: ' . " " . $_SERVER['REMOTE_ADDR'] . "\n";
fwrite($fp, $s_remote);

Then we append the $_POST['search_term'] variable to the file:

$s_remote= 'Search Term: ' . $_POST['search_term''] . "\n";
fwrite($fp, $s_remote);

and close the file until it is needed again:

fclose($handle);

...and that's all there is to it.

Using an include statement, such as:

include("gather_data.php");

We will insert the above script into the page(s) where we want to collect the specific data.

I hope this simple script gives you enough background to play with logging specific data on your website. Please give it a try using some additional variables.

Best of luck.

[kgun]

Originally Posted by Tech Manager

If this topic garners enough interest I will expand it into some more advanced logging procedures including security responses.

Great info, what about browser statistics?

[Tech Manager]

Browser statistics are as easy to obtain as any other data. But, keep in mind, during a hack attempt browser information can be masked, hidden or spoofed in the hacker's script. But in general your browser info is fairly accurate.

You can modify the script above to include the following lines to write to the text file:

$s_user= 'HTTP_USER_AGENT: ' . $_SERVER['HTTP_USER_AGENT'] . "\n\n";
fwrite($fp, $s_user);

The $_SERVER['HTTP_USER_AGENT'] will glean the specifics on the visitor's browser.

[Tech Manager]

As an added note, if I am gathering copious data I am probably not going to rely on a flat file (text file) to store the data. I will usually store data in a customized database so I can review live data on the fly, generate reports, graphs, etc.

[kgun]

What about an XML driven CMS / native database? Three obvious advantages:

1. SEO friendly if it is well (folder) structured. Content not hidden in a Database.
2. No need for a database platform. Sun Buys MySQL For $1 Billion
3. Fully compliant with the XML family of technologies.

[Tech Manager]

It's all a matter of personal choice. Use what you believe to be the best for your situation.

[wige]

Originally Posted by Tech Manager

As an added note, if I am gathering copious data I am probably not going to rely on a flat file (text file) to store the data. I will usually store data in a customized database so I can review live data on the fly, generate reports, graphs, etc.

What type of database do you find most efficient for storing the records? Do you use a seperate light database app on the server, or a simple table under your existing database engine, or some other type of solution?

It sounds like whatever solution was implemented would have to be very low on overhead so that new entries could be created with minimal resource impact, overhead from dynamic solutions such as relational features would be eliminated to minimize total file size, but enough power to collate the data for searching and reporting functionality. It would also have to be very efficient for handling a narrow table with few fields but many records.

[kgun]

Two related articles:

Typical Programmer - Why Programmers Don’t Like Relational Databases

Beyond Search » Search, Content Processing, and the Great Database Controversy

[Tech Manager]

Originally Posted by wige

What type of database do you find most efficient for storing the records? Do you use a seperate light database app on the server, or a simple table under your existing database engine, or some other type of solution?

It sounds like whatever solution was implemented would have to be very low on overhead so that new entries could be created with minimal resource impact, overhead from dynamic solutions such as relational features would be eliminated to minimize total file size, but enough power to collate the data for searching and reporting functionality. It would also have to be very efficient for handling a narrow table with few fields but many records.

Wige:

You ask some great questions and offer terrific follow up comments.

I don't think a single-solution answer would be adequate. When I am consulting with a client I will make recommendations that are tailored based on the needs, desires, goals, technical resources and budget of the individual clients.

Low overhead is almost always a goal except in certain very rare situations. Sometimes it is enough to add a single table containing a few fields and have no other relationship to the data other than auto incrementing a primary key. I have several clients who use just such a simple design and then run additional analytical programs on the backed up database.

Some clients want something a little more multidimensional. They want a much more robust solution including better relationships between the relative data.

Many of my dedicated server clients rely on nothing more than analysis of their access and error logs using a third party log analysis program.

For your typical webmaster on a shared hosting platform, where server logs may or may not be available, a flat file or very simple database may be sufficient.

For me, as I do some security consulting, I prefer to log everything. This helps me stay on top of trends, exploits, vulnerabilities, etc., and not just focused on marketing related data. I do my best to design my scripts and databases for ultimate efficiency with very low overhead.