Re: Assessment tools/Scanners

[stuart () SiliconDefense com (Stuart Staniford-Chen)]

Greg Shipley wrote:
> >         Why is it so difficult to develop an evaluation criteria which can
> > rate IDS packages in terms of which can (a) effectively generalize from
> > known exploits in order to place alarms on similar but not identical
> > attacks, and (b)  alarm areas of potential vulnerability, even if no exploit
> > has yet been published?
>
> Welp, like I think Dug Song touched on, you would need to agree on, at a
> minimum:
>
> 1) a standardized and universally accepted list or DB of known
> vulnerabilities.
> 2) a set of tools to test/exploit those vulnerabilities

I'm not sure this is enough for a truly useful evaluation.  You also need 

3) knowledge about how likely a vulnerability is to actually get exploited.

IDS's either don't keep up with fast networks at all, or just barely keep up
under the right circumstances.  Every signature adds work that the IDS has to
do, and makes it less likely that it will keep up.  So you probably don't
want a product that will detect absolutely everything.  You want a product
that will detect things most likely to be thrown at you.  And thus in a
useful real-world evaluation, you should not be counting total
vulnerabilities detected, you should be weighting the score by likelihood of
vulnerabilities actually being exploited.

Unfortunately, "probability that attack technique Y will be used in a
randomly chosen attack somewhere in the world today" is a little hard to
measure:-).  And, like the list of all attacks known, it changes all the
time.  That's why doing IDS evaluation is difficult.  (Which is not to slight
the efforts of Lincoln Labs and others who are doing them; it's much better
to have some results than none).

Stuart.

-- 
Stuart Staniford-Chen --- President --- Silicon Defense
                   stuart () silicondefense com
(707) 822-4588                     (707) 826-7571 (FAX)