Intrusion Detection Systems: Re: IDS Testing (WAS: Re: kernel implementations)

["Talisker" <Talisker () networkintrusion co uk>]

Archive: http://msgs.securepoint.com/ids
FAQ: http://www.ticm.com/kb/faq/idsfaq.html
IDS: http://www-rnks.informatik.tu-cottbus.de/~sobirey/ids.html
HELP: Having problems... email questions to ids-owner () uow edu au
NOTE: Remove this section from reply msgs otherwise the msg will bounce.
SPAM: DO NOT send unsolicted mail to this list.
UNSUBSCRIBE: email "unsubscribe ids" to majordomo () uow edu au
-----------------------------------------------------------------------------
----- Original Message -----
From: "Greg Shipley" <gshipley () neohapsis com>
To: <ids () uow edu au>
Cc: "Dug Song" <dugsong () monkey org>; <FOCUS-IDS () securityfocus com>
Sent: Monday, July 24, 2000 2:08 AM
Subject: IDS Testing (WAS: Re: IDS: kernel implementations)


> - Speed
> - Accuracy
> - Response Latency
> - Overhead
> - Noise
> - Stimulus Load
> - Variability
> - Usability
Greg
I like your criteria a lot I'd like to add a few things that are down on the
list of importance

Whilst I hate to use the C word, how about Cost, I often see this as the
bottom line and have to compromise most other things to meet it.  Much as I
want to recommend a Rolls Royce solution, if the budget will only stretch to
a Trabant then thats all they can have.

Presentation of alerts HTML seems to be becoming more popular, call me old
fashoned but I like the way a GUI frontend can put more data on the screen

Ability to write ones own signatures

Theres a few other things but I'm already starting to sound like a wish
list. I put a "possible Features" list together some time ago and put it on
the NIDS page, some of the things seem a little lame now in retrospect,
but.....
www.networkintrusion.co.uk net ids (the direct link isn't working at the
moment)

> While some of those terms are a bit vague without hearing the explanation
> behind them, let me toss out some of my own ideas and you guys can rip
> them apart: (a lot more fun having a live target, yeah?)
>
>
> 1. Capacity/Scalability - this is one of the perceived big points: How
> much can it handle.  Until some demographics come out on actual NIDS
> deployments, I can only base the importance of whether NIDS (A) can handle
> 150Mbps of traffic vs. NIDS (B) on personal observations and "the word on
> the street."  But, regardless, it *IS* an issue and it has to be looked
> at.  Also, "scalability" can also be looked at as not only how much
> bandwidth the NIDS can handle, but how many sensors the management
> platforms can handle, how many alerts can be processed, etc.  Huge can of
> worms....but more points to cover.  Also, the footprint of a host agent on
> the host-side...
>
> 2. Accuracy - need to address false-positives, thoroughness and ACCURACY
> (nasty) of signature base, thoroughness/completeness of the engine (i.e.
> addressing all of Ptacek's complaints and the "fragrouter" test), etc.
>
> 3. Diversity/Platform Coverage - How many platforms does the product
> support?  Not an issue for NIDS, but a BIG issue for host-based ID
> systems.  (as a side note, according to Mr. Maxion's paper, the use of the
> term "host-based" would be considered a "murky definition" *smirk*)
>
> 4. Manageability/Complexity - more then a few people on this list have
> mentioned that we should consider the "human" factor in this.  For
> example, while products like Dragon are quite thorough, throwing Dragon at
> a bunch of NT admins isn't going to fly very well.  Also, some of the
> management consoles, IMHO, suck ass (that's a technical term, BTW).  So we
> need to measure/investigate how hard these things are to deploy, maintain,
> and generally use....both the engines AND the management platforms.
>
> 5. Timeliness - this one, funny enough, no one seems to talk about.  How
> long till I get new signatures?  Given, most IDS deployments are used as
> reactionary tools.  That is, event X occurs and the IDS does Y (alert,
> page, shun, whatever).  However, IMHO you still want to be at least
> SOMEWHAT current in what you are looking for.  In talking to the guys that
> drive the SANS "Secure Alert Consensus" service (we work in the same lab)
> they are reporting between 10-30 new vulnerabilities PER WEEK.  Now, not
> all of those can be coded into IDS sigs, HOWEVER, if a vendor only
> releases sigs quarterly, you could be behind on 200+ some attack
> signatures EASILY. Then there is also the method of updating, too.
>
>
> These are all product issues.  There are also issues surrounding cost,
> vendor stability, the vendor's support network, etc., but I'd like to
> focus on hammering out the PRODUCT side of this right now.
>
> Ok, so based on those points - what else is there to add from a 10,000ft
> view?  From here we can go into how you actually TEST these things -
> that's the nasty part, but IMHO we need to start here.
>
> Comments?  Thoughts?
>
> -Greg
www.networkintrusion.co.uk
                    '''
                 (0 0)
  ----oOO----(_)----------
  | The geek shall        |
  |  Inherit the earth     |
  -----------------oOO----
               |__|__|
                  || ||
              ooO Ooo


The opinions contained within this transmission are entirely my own, and do
not necessarily reflect those of my employer.