Home page logo

basics logo Security Basics mailing list archives

RE: No Budget Static Log Analysis
From: Dave Kleiman <dave () davekleiman com>
Date: Fri, 27 Jul 2012 12:21:12 -0500

Hello Steve,

There are a wide variety of excellent log  tools, Logstash, Splunk. Sawmill, etc. as previously mentioned.  If you 
decide to use Log Parser you can create a toolbox of scripts and have them ready:

Some like:

SELECT time,
INTO ISAlertByIP.csv
FROM ex*.log
       (cs-uri-stem LIKE '%ALERT%')
       (IPV4_TO_INT(c-ip) BETWEEN IPV4_TO_INT('') AND IPV4_TO_INT(''))
GROUP BY c-ip,time,date,cs-uri-stem

Save as IISAlertByIP.sql  and call it from the logparser dir.  Logparser file: IISAlertByIP.sql  

I have a bunch of scripts you might find handy on my site:  

Feel free to ping me with any questions,


Dave Kleiman - http://www.ComputerForensicsLLC.com - http://www.computerforensicsexpertwitnesses.com/

4371 Northlake Blvd #314
Palm Beach Gardens, FL 33410

-----Original Message-----
From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Stephen Mullins
Sent: Wednesday, July 25, 2012 15:34
To: security-basics () securityfocus com
Subject: No Budget Static Log Analysis


I am involved in a project that performs analysis of a static set of logs provided in ASCII/plain text format looking 
for signs of malicious activity using lists of known "indicators" (IP addresses, domain names, user agent strings, 
etc.).  The logs can be from any number of devices (firewalls, web proxies, DNS servers, etc.) and can be 
formatted/delimited in whatever format is native to the device that generated the logs.  The smallest set of data 
received thus far was
200 gigabytes and the largest 2.3 terabytes.

So far I've be slogging through the logs with *nix tools, shell scripts, perl, etc.  But this takes a lot of time, 
doesn't scale, and requires custom work for each new set of logs.

I have no budget to procure a commercial solution like Splunk, ArcSight, Saw Mill, etc.  I do have a couple of 
relatively decent servers and some Network Area Storage devices to work with.  I've done some research into "free" 
solutions and come up with the following short list:

Log Parser (Windows)

Has anyone tackled a similar problem in the past?  If so, do you have any advice or recommendations?  Of the "free" 
solutions researched the first one I am inclined to try is LogStash.


Steve Mullins

Securing Apache Web Server with thawte Digital Certificate In this guide we examine the importance of Apache-SSL and 
who needs an SSL certificate.  We look at how SSL works, how it benefits your company and how your customers can tell 
if a site is secure. You will find out how to test, purchase, install and use a thawte Digital Certificate on your 
Apache web server. Throughout, best practices for set-up are highlighted to help you ensure efficient ongoing 
management of your encryption keys and digital certificates.


  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]