Ok, I'll give this a go. Like Sam, I used to work for an
IDS vendor (*chuckle*), and did some work in this area during
my sentence there.
However, I've not been "in the industry" or even paying much
attention to it for awhile, so with that in mind, please observe
as I try to walk the wire without falling into the pit below...
From Marty's first post:
>1) Detect, Attack Present, Vulnerable: True Positive
>2) Detect, Attack Present, Not Vulnerable: Nontextual (i.e. detect
>requiring contextual data to resolve)
>3) Detect, No Attack, [vuln|not vuln]: false positive
>4) No Detect, Attack Present, Vulnerable: False Negative
>5) No Detect, Attack Present, Not Vulnerable: ?
>6) No Detect, No Attack, [vuln|not vuln]: Don't care (true negative?)
I agree with Case 1.
I grok the terminology of Case 2 but in my days as an analyst (pre-IDS
vendor),
I'd consider Case 2 something I'd very much want to know about. The presence
of an attack, regardless of its lack of impact against the monitored
network, is still
valid information and usable.
I agree with Cases 3, 4, and 6.
Using my logic (addled as it may be) from Case 2, Case 5 is also a False
Negative.
It's information I wanted to know about, but wasn't told of.
At 09:57 PM 10/23/2003, Marty wrote:
>On Oct 23, 2003, at 6:53 AM, Sam f. Stover wrote:
>>On Wednesday, October 22, 2003, at 11:22 PM, Martin Roesch wrote:
>>In case 2 the "nontextual" isn't a false positive but I think that most
people are calling it an FP these days. I >>*personally* think that's a
misconception. What we have in that case is a *real attack* that your IDS
is detecting >>exactly as it was asked to. Just because it doesn't have
the additional information about the context or relevance >>of the event
isn't a problem with the IDS, it's a side effect of the way that NIDS have
been built for the past 10 years.
>In the not too distant past I would have agreed with this - but I think
as IDS implementations grew, the way people >describe FPs has changed. I
think today's IDS *needs* to know "the additional information about the
context and >relevance" - because the event you are referring to is what
I'll call an "effective FP". Effective because any time I >spend trying to
track down an IIS attack on an apache box is wasted effort. I completely
understand your point >Marty, because an attack did occur, and the IDS did
log it. However, if it is going to log it, then I want it to tell me >that
the severity of the attack is lessened because it didn't succeed. Even
better, I want to see the 404 or 403 error, >so I can show my boss why I
didn't even bother to look into it.
>>I agree and we're building new technology at Sourcefire to address it,
and I know other companies are either >>building new stuff or adapting old
stuff (e.g. vuln scanners) to get the same sort of data. I was just
pointing out that >>systems like Snort as it currently exists don't have
this notion built into them so saying that alert verification >>reduces
false positives when it's actually addressing nontextuals is something that
I want to discuss, especially >>since it's my code that's producing the
"false postives". :)
Oh ok, Sam already expressed my Case 2 - oops. *a growl emanates from the
pit below...*
>I want my IDS to differentiate between an IIS attack on my apache box and
an IIS attack on an IIS box. I don't really >care how it does it.
>The two main methods, as I see it, are passive fingerprinting or
integration with another tool like a vuln scanner. >Both have their
drawbacks w/ relation to different environments - which could probably fuel
a complete thread.
>>Yup.
>The IDS landscape has changed. Ten years ago, the type of event
mentioned was probably not considered a FP. >But at that time, IDS was an
infant and people weren't dealing with events on the scale of millions per
day like they >are today. Current-day NIDS need to evolve to solve the
problems that current-day users are facing. IMHO 10 years >ago, NIDS
administrators could afford to be a bit more interested in all kinds of
attacks. IDS was a new and exciting >technology. I think it's lost some
of it's glamour since then and people have to use it as just another
tool. And the >people I talk to don't have the time nor resources to run
down half of the "real" attacks, much less look into attacks >that will
never succeed.
>>Yes. Separating the wheat from the chaff is becoming increasingly
important in IDS as we all know, I'll be >>interested to see how the
different techniques and approaches people are using to address this
problem actually >>work in production.
I had the chance to see some tools being used to address this topic
a few months ago, working on a ton of live data.
Andrew Hall's post discusses it in adequate detail, and boiling it out,
it comes down to associating events from as many sources as may be
available into its correct aggregated identity. Current SIMs can do that,
but they are only as good as their rulebase and the validity of the data
being fed them. Still, they've come a long way from the last time
I had seen one.
Pretty much everything in the data feed chain can false, however, so
you need a SIM with enough smarts to be able to say something along
the lines of, "I didn't get everything I expected to see to satisfy this
condition,
but I got enough good stuff to go ahead and throw a flag, just to be safe."
In other words, a variation on my Case 2 logic above. With the "Devil" in
that sentence being the word "good". Computers can be incredibly bad
at identifying "good".
But I did take a shot at the problem during my incarceration in IDS-vendordom,
and came up short. Collaborating with a fellow detainee, we found that "good"
could be defined relative to one source of data - an IDS in particular -
but only
in the context of the validity of the IDS' own output. Where I failed was in
not provisioning for other relevant data sources - trending, VA, asset
assessment,
etc. It was a good idea, just not enough of it, and not thought through
well enough.
I can chalk some of that up to living inside the "glass bubble" effect Sam
describes.
The rest is mostly likely one of the definitions of Wu Li from Gary Zukav's
book.
I don't expect an IDS to be a one-stop shop for all relevant data and I doubt
any permutation of it ever will be. SIM seems to be the right idea, but it's
not mature yet. It wasn't that long ago we were saying the same thing about
IDS,
so I'm not overly concerned about SIMs - they're very much worth using
right now, just as IDS' were then. One way or the other, I'm sure we'll
get it all nailed down eventually...which will happen just before the
Universe goes
foom and forces us to start over again from scratch...
-Marty
Randy
-----
"Nor does it do anything to make lemons bigger or encourage owls to explode."
--- MartinG on /. ---
---------------------------------------------------------------------------
Network with over 10,000 of the brightest minds in information security
at the largest, most highly-anticipated industry event of the year.
Don't miss RSA Conference 2004! Choose from over 200 class sessions and
see demos from more than 250 industry vendors. If your job touches
security, you need to be here. Learn more or register at
http://www.securityfocus.com/sponsor/RSA_focus-ids_031023
and use priority code SF4.
---------------------------------------------------------------------------
Received on Oct 24 2003
Intro
