mailing list archives
Re: RE: IDS (was: FW appliance comparison)
From: Brian Loe <knobdy () gmail com>
Date: Mon, 30 Jan 2006 09:10:03 -0600
On 1/27/06, Marcus J. Ranum <mjr () tenablesecurity com> wrote:
Joking aside, I think you got my point. It's certainly possible to
handle the kind of logging load you're describing. I wouldn't go
quite for far as to say it's "easy" - but your original post made it
sound like you felt it was a near impossibility, or something like that.
I'll reserve opinions of possibility for when I find out how to use the data...
But you don't have to convince me
It _sounded_ like I do. After all, you were the one who said:
"I'm not sure how we could do it"
That's a whole different situation from if you'd said "I have to
convince my management not to be stupid" in which case I
would have entirely ignored your posting. After all, "I have to
I suppose, but where I was coming from was that this requires a great
deal of resources - not only in space - and that requires money. So
now I don't just have to convince him of the need for doing such but
justify the associated costs. Naturally its slightly political, but
only because I would have to put the fact that we have our pants down
right now pretty delicately.
So, they obviously understand at least some of the value of
collecting such information - otherwise they would have it
completely turned off "for performance reasons" or something
Performance is certainly not an issue, but the only use they get from
it right now is the ability to fire up their desktop based viewer and
watch a problem occurring real-time. That's nice and helpful for
troubleshooting, to be sure, but does nothing for what you have
So if you're working for a company so stupid that they'd rather
spend $10,000 for a "device" from Cisco (or whoever) or $100,000
for something from IBM (or whatever) then none of us can help
you and your best strategy is to crawl back under a rock and go
back to hoping the sky doesn't fall on your head. There's nothing
any of us on this list can do to help you; you should be talking
to a Cisco sales-rep, instead.
But - back to your assertion that I'm saying that "they have to
spend money" - wait a minute: didn't you already say you had
some hardware? Presumably a computer with a hard disk, right?
Maybe you don't have a terabyte in the darned thing but if you
managed to do something useful and demonstrate some value
to it, you'd probably find that they could afford a hard disk
upgrade. But what do I know?
But wait, isn't that what I'm asking - how to go about doing just
that? They've tried, I've tried, to do the logging - and always with
something running on the data to help make it useful. You say it can
be done and that it is useful - I'm just wanting to hear how. I think
generally the mindset has been to log for troubleshooting purposes,
not security (and this should probably include my last effort).
This has nothing to do with stupidity, btw, but with ignorance.
So, it sounds to me like you jumped into the problem
without actually thinking about it, first, and failed. That
should have come as no surprise. But it appears that
you generalized that failure into a theory that "it can't
be done." Um, no.
I think the comment was I don't see how it could be done. I played
with it as best I could before - drilling for what I thought I ought
to review on a regular basis - but it didn't work. Obviously, having
tried, I didn't see how it could be done. Still don't. The problem
seems to get exacerbated, I might add, if you're saving all the data
(no drilling) and reviewing it all manually, looking for something
you've never seen before and can find no reference for.
Without a system
to at least *help* you analyze it you're simply swimming in quicksand,
flailing in fact.
The last log analysis project I got dug into (Hi Paul!) we used
really advanced tools like "grep" and "more" and "guinness stout"
and figured out what the data looked like, then figured out what
we wanted to do with it. Then I wrote a few carefully tailored
500 to 2000 line-of-code utilities in C that did the job, and
let them run for a day and *poof* we found stuff! By the way,
since I had about 40 gigs of bzipped log data that I was trying to find
a single unknown event in, I had to take into account the speed
of the various tools and do some back of the envelope math,
first. If I'd used a database, we'd still be sitting waiting for
results - and the project was 2 years ago.
If only every company could afford someone such as yourself...
As for IDS, I
personally think its a mostly useless tool - especially the way they
have it implemented here.
But you're the guy who said:
"But, on the bright side, our 2k IDS system did
eventually begin blocking it from all but one customer site."
comparing your "$250k" log analysis system to your $2k IDS -
which certainly doesn't make it sound like you think your
IDS is useless. Make up your mind, would you?
Did I say that? Can't recall the context now, but I'm thinking that I
was probably discussing an already known issue that IDS eventually
managed to block, after getting the appropriate signature. That's not
what I would consider an IDS boast.
Remember, though, we're talking about my crunching through
more than 100 times (decompressed) as much data as you were
complaining about, in less time than it was taking you to
collect the amount you were complaining about. That tells
me your problem is probably highly solvable.
As I said, that was one firewall, we have around 8. And, as most of
what I just snipped out shows, I'm at a loss for how *I* can do it
better still since I'm not a developer, still have no idea what I'd be
looking for...etc.,etc.. I am currently tracking things like processor
and memory loads, in hopes that a jump in activity - especially at odd
times - would be an alert of something bad happening. Since you
mentioned it, I might see if I can better monitor numbers of log
entries as well.
Since your point about the data being needed after the fact I'm going
to be pushing to get a place for us to store our logs - then, at the
least, if something should happen we can get someone to figure it out
Something that seems to be lost here though is that when I was saying
a Gig an hour, that was in compressed format. The logserver was
rolling the logs at a gig. Not only is that a lot of data to store but
that's a lot of data through the NIC. Multiplying that by two is going
to be interesting, let alone 4 or more.
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
RE: RE: IDS (was: FW appliance comparison) Jon Czerwinski (Feb 02)