Home page logo

firewall-wizards logo Firewall Wizards mailing list archives

Re: parsing logs ultra-fast inline
From: "Marcus J. Ranum" <mjr () ranum com>
Date: Tue, 07 Feb 2006 14:52:38 -0500

Chuck Swiger wrote:
Without fighting too hard, many log analysis tools for
things like webserver or squid or firewall rules seem to process ~10K lines or
events per second, which works out to a gigabyte every ten minutes or so,
whereas other tools seem hopelessly incapable of handling large data sets.

I think it's because a lot of webserver analysis tools are designed to
rip through the data and provide statistical summaries and sorted
hit-lists, whereas the security-oriented log processing tools are
aimed at audit functions. Since the security problem is less well-bounded
than "show me the top 50 pages on my site!" the designers of those
systems often reach for the biggest hammer in their toolbox and
stuff everything into a SQL database, which promptly falls over,
leading them to conclude "it can't be done."

As we discussed last week; if you put some thought into figuring out
what you want to get from your log analysis, you can do it at extremely
high speeds, pre-compute all the running totals you need, cache
views into the data-sets as necessary, etc. But I used that evil
phrase "put some thought into..." and we know that most IT managers
would rather buy a $20,000 thingamajig than "put some thought into..."


firewall-wizards mailing list
firewall-wizards () honor icsalabs com

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]