IDS Testing (WAS: Re: kernel implementations)

[Greg Shipley <gshipley () neohapsis com>]

Archive: http://msgs.securepoint.com/ids
FAQ: http://www.ticm.com/kb/faq/idsfaq.html
IDS: http://www-rnks.informatik.tu-cottbus.de/~sobirey/ids.html
HELP: Having problems... email questions to ids-owner () uow edu au
NOTE: Remove this section from reply msgs otherwise the msg will bounce.
SPAM: DO NOT send unsolicted mail to this list.
UNSUBSCRIBE: email "unsubscribe ids" to majordomo () uow edu au
-----------------------------------------------------------------------------


On Sat, 22 Jul 2000, Dug Song wrote:

> oh, but vendors will simply claim that they're "the fastest, most
> accurate, and most widely interoperable" in the absence of any hard and
> fast criteria. just look to the firewall market for precedent.
>
> without well-defined quality metrics, who's to say for certain how any two
> IDSs compare? what you measure, and how you measure it, are of the utmost
> importance when evaluating a system - but we haven't even begun to develop
> test methodologies that are generally useful.
>
> but we've been over all this before.
>
>       http://msgs.SecurePoint.com/cgi-bin/get/ids-9910/9/1/1/1/2.html


Ok, as I've been re-tooling the lab to handle some more IDS testing I've
taken good notes on some of the wonderful discussions recently, and went
back and looked at some of the suggested URLs.  While it is entirely
possible that I am missing the obvious, I fail to see why some of these
references are held in such high-regard.

First off, the Los Alomos paper referenced here:
http://www.anzen.com/news/anzen_chart.pdf

I don't get it.  This is a thorough chart of features - great.  Is there
*ANY* testing behind this report, what-so-ever?  ANY?  Is it me, or is
this just a glorified features matrix?  I can't believe that the people on
this list who are SO CRITICAL of testing methodology would give such
weight to a paper that has ABSOLUTELY NO TESTING behind it.

Is it somewhat useful?  Yes - but someone, please, fill me in here!

Second, the paper referenced here:
http://www.zurich.ibm.com/pub/Other/RAID/Prog_RAID98/Program.html - from
Roy Maxion at RAID98.  Now, this one is a bit better, but I still don't
understand what makes this one stellar, either.  Or is it just that there
isn't anything considered "good" out there?  While I respect the author's
time and effort, and there are some good points made, I still fail to see
how this is ANYWHERE CLOSE to providing a framework for IDS testing...or
even CLOSE to addressing the issues at hand.  This appears to be more
focussed on maintaining scientific method then it is in addressing IDS
testing. (...and talking about "Planck's Discovery of the law of blackbody
radiation" and "Fleming's discovery of penicillin", etc. - BTW, what is
"Cinnamon?")

Again, if I'm missing something here, by all means, fill me in - but I'm
struggling here.

So let me back up a second.  Since this list is populated by all of the
big players, and everyone insists that there is really poor testing and
comparisons going on these days, why not hammer those issues out here?  
Why not come up with a framework for testing and comparisons?

Let me state this simply: I, as well as many other people on this list, am
testing these products - what should I be looking for?  (I obviously have
my own ideas, and some people have chimed in here and there - but I want
to hear it from the people complaining)  Mr. Maxion talks about "Figures
of Merit for an Intrusion Detection System" as being:

- Speed
- Accuracy
- Response Latency
- Overhead
- Noise
- Stimulus Load
- Variability
- Usability

While some of those terms are a bit vague without hearing the explanation
behind them, let me toss out some of my own ideas and you guys can rip
them apart: (a lot more fun having a live target, yeah?)


1. Capacity/Scalability - this is one of the perceived big points: How
much can it handle.  Until some demographics come out on actual NIDS
deployments, I can only base the importance of whether NIDS (A) can handle
150Mbps of traffic vs. NIDS (B) on personal observations and "the word on
the street."  But, regardless, it *IS* an issue and it has to be looked
at.  Also, "scalability" can also be looked at as not only how much
bandwidth the NIDS can handle, but how many sensors the management
platforms can handle, how many alerts can be processed, etc.  Huge can of
worms....but more points to cover.  Also, the footprint of a host agent on
the host-side...

2. Accuracy - need to address false-positives, thoroughness and ACCURACY
(nasty) of signature base, thoroughness/completeness of the engine (i.e.
addressing all of Ptacek's complaints and the "fragrouter" test), etc.

3. Diversity/Platform Coverage - How many platforms does the product
support?  Not an issue for NIDS, but a BIG issue for host-based ID
systems.  (as a side note, according to Mr. Maxion's paper, the use of the
term "host-based" would be considered a "murky definition" *smirk*)

4. Manageability/Complexity - more then a few people on this list have
mentioned that we should consider the "human" factor in this.  For
example, while products like Dragon are quite thorough, throwing Dragon at
a bunch of NT admins isn't going to fly very well.  Also, some of the
management consoles, IMHO, suck ass (that's a technical term, BTW).  So we
need to measure/investigate how hard these things are to deploy, maintain,
and generally use....both the engines AND the management platforms.

5. Timeliness - this one, funny enough, no one seems to talk about.  How
long till I get new signatures?  Given, most IDS deployments are used as
reactionary tools.  That is, event X occurs and the IDS does Y (alert,
page, shun, whatever).  However, IMHO you still want to be at least
SOMEWHAT current in what you are looking for.  In talking to the guys that
drive the SANS "Secure Alert Consensus" service (we work in the same lab)
they are reporting between 10-30 new vulnerabilities PER WEEK.  Now, not
all of those can be coded into IDS sigs, HOWEVER, if a vendor only
releases sigs quarterly, you could be behind on 200+ some attack
signatures EASILY. Then there is also the method of updating, too.


These are all product issues.  There are also issues surrounding cost,
vendor stability, the vendor's support network, etc., but I'd like to
focus on hammering out the PRODUCT side of this right now.

Ok, so based on those points - what else is there to add from a 10,000ft
view?  From here we can go into how you actually TEST these things -
that's the nasty part, but IMHO we need to start here.

Comments?  Thoughts?

-Greg