Re: IDS thoughts

[Mike Frantzen <frantzen () nfr net>]

> You are joking, right ? There's a whole lot of research still open in the
> IDS field. Just to begin, you are apparently forgetting that there's a whole
> paradigm of ID, anomaly-based detection, which has just been forgotten by
> the mainstream development.

I don't think anyone has forgotten anomaly-based detection.  Most
players are taking a hybrid approach.
 
> In the next few years, while established IDS products will strive to keep up
> to date their growing signature base, and face increasing performance
> problems, probably some attention will be returned at that preliminary
> choice of matching bad_things instead of good_ones.

Keeping up isn't as hard as you would think.  Even for pure pattern
matching, most algorithms scale roughly on the order O(log n) w.r.t
performance and number of signatures.  Some of the poorer algorithms
start getting ugly cache/tlb/bp behavior but there are algorithms that
fix any two out of those three secondary affects.

> When it comes to firewalling, we all agree: you just shut down everything
> very tight, then open up what few ports you actually need. When it comes to
> privileges and authentication, we do the same thing, and we are quick to
> point out the error when someone tries to filter out unwanted input, instead
> than specifying what is the expected one.
> Oddly, when we talk about IDS and antivirus software, we blindly accept that
> there's only one way to do it: by describing what we do NOT want on our
> system by the mean of signature. Well, this happens to be a BAD idea, even
> if until now it has given us some satisfactions.

Ok.  I do both firewall development (OpenBSD) and IDS development (NFR).
And they are totally different, dare I use a buzz word, paradigms.  To
give a fairly deterministic measure of the difference, the size of NFR's
regression tests alone are four times larger than the entire OpenBSD PF
firewall.  With firewalls, you have the luxury of being able to assume
port == protocol (yes yes, you can do very rudimentary protocol
validation with things like inspect scripting; and there are a few
proxying firewalls that checkpoint hasn't killed off yet)

That naivety from the security device just doesn't cut it in the IDS
space.  Sit down and stare at several captures of HTTP transactions.
Ones from IE, Netscape, Konq, Opera....  They all look different and
this is where theory diverges from implementation.  An anomaly in one is
perfectly normal in the other.  It gets worse, the transactions start
looking differently depending on the server they talk to.  Sure, so we
can leave the client protocol models agnostic of the server type.  But
now you start to factor in that HTTP clients lie about what program they
are.  And certain venders are notorious for protocol variations between
minor patch levels...

Yes, pure anomaly detection is a very complex process and *VERY*
difficult to create something that doesn't have a propensity towards
false positives.  Subtle changes in an environments usage patterns often
occur suddenly and tend to come across as anomalies -- now the IDS/IPS
admin gets inundated after acclimating to relatively few alerts.

Thus you see many venders transitioning (have already done so) to doing
anomaly detection where feasible, and "bad thing" detection when not.

I'll make a standing offer, I will buy anyone a cookie that can describe
their enterprise network usage adequately enough that would allow pure
anomaly detction.  Hint, you control all the clients versions and all
their connections either terminate at your servers or your proxies.
 
.mike
frantzen@(nfr.com | cvs.openbsd.org | w4g.org)
PGP:  CC A4 E2 E8 0C F8 42 F0  BC 26 85 5B 6F 9E ED 28

-------------------------------------------------------------------------------
INTRUSION PREVENTION: READY FOR PRIME TIME?

IntruShield now offers unprecedented Intrusion IntelligenceTM capabilities 
- including intrusion identification, relevancy, direction, impact and analysis 
- enabling a path to prevention.

Download the latest white paper "Intrusion Prevention: Myths, Challenges, and Requirements" at: 
http://www.securityfocus.com/IntruVert-focus-ids2
-------------------------------------------------------------------------------