False positives, negatives and don't cares

[Martin Roesch <roesch () sourcefire com>]

Hey all,

Just following up the "IDS is dead, etc" thread, I thought I'd lay out 
some of my thoughts about data quality and event value coming out of 
NIDS.  Maybe some of you will find a little food for thought in the way 
I look at IDS output, this certainly isn't the only way to think about 
it.

First off we have the false negatives.  These are events that the IDS 
misses for whatever reason, whether it is actively evaded or misses an 
attack due to packet loss or improper traffic analysis.  These are 
annoying and IDS vendors spend great amounts of time making sure that 
they don't occur with any sort of frequency.  I'd probably be willing 
to wager that IDS vendors spend the lion's share of their time working 
on this side of things, which is why we have problems with data 
management today.

Next we have false positives.  False positives happen when the IDS 
detects what it's not supposed to, typically because of flawed traffic 
modeling or weak rules/signatures/anomalies specified.  Insertion 
attacks can also cause false positives when an attacker manages to get 
the NIDS to accept traffic that the end host will not and making the 
IDS' traffic model diverge from the target's.  This is the annoying 
stuff that gives NIDS a bad name, but I wonder what the real percentage 
of these is versus the next type.

The next group that I'll discuss is traffic that consists of real 
attacks heading to targets that aren't vulnerable to them.  These 
frequently are labeled as false positives, but in fact they're real 
attacks applied in contextually irrelevant ways.  This is something 
that I think is a bigger problem than most people have figured out, 
most everyone calls these false positives.  The way to reject the noise 
and increase the signal in this case is to contextualize the data 
coming out of the NIDS, apply what is known about the network and its 
vulnerability picture to the data.  This lets you ignore the events 
that don't correlate to any known vulnerabilities or that are heading 
to machines that aren't capable of being vulnerable to a given attack 
thereby greatly increasing the value of the data coming out of the NIDS.

This is one of the things that frustrates me when (some) people say 
that Snort generates "tons of false positives".  If you know how 
Snort's detection engine works, you know that Snort merely presents a 
framework run through a set of analysis nodes arranged in certain ways 
against a network traffic stream (or single packet).  Snort doesn't 
have a hard-wired detection engine, it is defined and built at run-time 
based on the user-defined rules that are loaded into it.  In this model 
it is very difficult to have false positives except in the case there 
are insertion attacks, Snort only detects what you tell it to.  If 
Snort detects a CodeRed attack heading to your Linux web server, it 
detected the real attack and let you know about it, *just as you asked 
it to*.  Just because that event notification isn't contextually 
relevant in your environment doesn't make it a false positive, your 
available courses of action are clear.  You can either modify the rule 
to tie it to IPs running a vulnerable version of IIS on win32, disable 
the rule, or provide proper prioritization for the data in post 
processing.  If you happen to have a target-based IDS, it could even do 
the auto-prioritization in real-time.

Lots of vendors are taking a stab at building the necessary software to 
apply this sort of context to that data coming out of NIDS 
automatically, I have my own favorite methods for doing it as well 
(obligatory Sourcefire plug here).  I think that the data that ends up 
on the "cutting room floor" after this contextualization process still 
has value for trending purposes and for the cases where we need the 
data to back track a real attacker in a more forensically intensive 
way.  If the right kind of data is collected and given to the NIDS 
sensor itself the false negative and positive problems can be addressed 
too...

This why I believe that the Gartner guys are clueless, they don't even 
have a conceptual framework in which to define the problem that they're 
complaining about so they're obviously unable to conceive of a 
solution, ultimately leading them to their laughable conclusion that 
"IDS is dead".

Hope some of you found this useful!

     -Marty

--
Martin Roesch - Founder/CTO Sourcefire Inc. - (410) 290-1616
Sourcefire: Enterprise-class Intrusion management built on Snort
roesch () sourcefire com - http://www.sourcefire.com
Snort: Open Source Network IDS - http://www.snort.org


---------------------------------------------------------------------------
Captus Networks - Integrated Intrusion Prevention and Traffic Shaping  
- Instantly Stop DoS/DDoS Attacks, Worms & Port Scans
- Automatically Control P2P, IM and Spam Traffic
- Ensure Reliable Performance of Mission Critical Applications
Precisely Define and Implement Network Security and Performance Policies
**FREE Vulnerability Assessment Toolkit - WhitePapers - Live Demo
Visit us at: http://www.captusnetworks.com/ads/31.htm
---------------------------------------------------------------------------