Re: RE: IDS (was: FW appliance comparison)

["Marcus J. Ranum" <mjr () ranum com>]

Cat Okita wrote:
Would you care to elaborate on the way that you handle the vast amounts
> of data that you collect, then?  Sorting the gold from the dross is a
> monumental challenge on a good day.


Use an artificial ignorance to weed out the majority of it, then
revector stuff that should be counted and quantified into a
round-robin database or something else you can do some
statistics with. Everything that comes through the back of
that process is worth human review, which might result in
it going into the artificial ignorance stop-list or statistics
engine.

For truly huge amounts of log data, you can use hardcoded
tools and get amazing data rates out of them; for example,
building a parse-tree out of nested calls to sscanf using the
magic %n operator to offset directly into the last match.

System log processing remains a backwater in spite of the
recent interest in the topic thanks to HIPAA and whatnot.
www.loganalysis.org has some resources on some of
this stuff. But it remains the land of do-it-yourselfers
because log data is very site-specific. On the other hand
it's not freakin' rocket science; if you just sit down and
start eyeballing the stuff you'll get an idea what you
need for your site within an hour or 2.

mjr. 

_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards