Home page logo

firewall-wizards logo Firewall Wizards mailing list archives

Re: parsing logs ultra-fast inline
From: John Adams <jna+dated+1139879645.1e269a () retina net>
Date: Wed, 8 Feb 2006 17:14:04 -0800 (PST)

I wrote a PIX log analysis tool awhile back which isn't extremely good, but it might give you a good start. It sucks the log file into a mysql databse, and then allows you to use a PHP based interface to browse it and get statistics.


It's old, though. I haven't worked on it in quite some time.

On Wed, 8 Feb 2006, Marcus J. Ranum wrote:

Brian Loe wrote:
Picking on me again already! Sheesh...

Nope, actually I'm picking on a superclass of companies and individuals
among whom you are an individual member. It's nothing personal! :)

Still have no idea, really, how to
configure syslog-ng and write a perl script as described - but I'll
fumble through it.

Googling for "parse pix log script" returns me 380,000
possible references and the first 3 look immediately promising.
Googling for "parse AIX log script" returns me 314,000
possible references and the first page has about 4 items that
look promising.

Question: Better to do it inline or off-line (for starters anyway)?

For testing and getting things working, I'd say to collect
the data to a hard disk then use a secondary process that
runs against the data on the disk. Once you have all that
working then you can put things in place to rotate the data
out when you're done with it.

A typical approach to doing this would be to use syslog-ng
to separate the log messages into the different apps that
you want to deal with and then deal with them each in
separate scripts that assess that app's logfiles. Note that
syslog-ng is not exactly "lightweight" but as long as you
can resist the urge to try to stick this stuff into a database
you will probably be fine.

I figure it would require less
overhead to analyze individual files by type (and therefore similar

Yup! Basically, you're talking about using syslog-ng as that
first-level of your parse tree that breaks things into sub-branches
by application. Of course syslog-ng is a gigantic sledgehammer
of a chunk of software to do something that simple, but it's
easy and flexible, etc.

Second question: Hasn't anyone else ever written these scripts? You
would think they'd be pretty widely available

There's this awesome website called www.google.com you
really ought to check out!!! It's for finding things on the internet!
And it's free and it's really fast!


firewall-wizards mailing list
firewall-wizards () honor icsalabs com

J. Adams                                        http://www.retina.net/~jna

firewall-wizards mailing list
firewall-wizards () honor icsalabs com

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]