Re: Assessment tools/Scanners

[vin () shore net (Vin McLellan)]

        Ryan M. Ferris <ryan25 () wenet net> suggested twisting the arms of IDS
vendors to obtain their internal test suites to validate the effectiveness
of their signature library in identifying the known threats, exploits, and
attacks.

        Dug Song <dugsong () monkey org> replied:

> but the problem is that vendors aren't using or developing test suites
> that determine how their systems FAIL. this is evident from the ways we've
> found to trivially elude them.
 
        Would you mind elaborating on this point further, Dug?  

        (I've enjoyed your papers, btw. Thanks for the effort.)

        How does one develop a test suite that will identify the Failure
when an IDS module does not identify the threat behind a novel attack?  (Or
an old attack presented in a novel manner?)

        With a rule-based system, how does one go beyond the list of known
attacks to alarm vulnerabilities as well as known threats?

        (I have alway presumed that the need to identify areas of potential
vulnerability -- as opposed to known exploits -- was a major reason the
leading IDS vendors have hired or contracted with various gray-sombrero
hacker groups.  NFR is getting backend filters from Mudge and the lunar
luminaries of the L0pht;  ISS has its Xtrodinary X-Force;  and Axent has its
sharpshooter SWAT group.  Does this work? Have these groups or similar
gray-cloaked warriors contributed to the state of the art or pushed the
envelope?)

        Why is it so difficult to develop an evaluation criteria which can
rate IDS packages in terms of which can (a) effectively generalize from
known exploits in order to place alarms on similar but not identical
attacks, and (b)  alarm areas of potential vulnerability, even if no exploit
has yet been published?

        Has anyone tried to evaluate these product historically, to see what
percentage of new and novel attacks  have been caught because one or several
IDS packages reached beyond the list of known exploits and detected
anomalities? (Too early for this, maybe?)

        Do you know if anyone is doing this now to track the _future_
success or failure of these IDS packages in identifying novel attacks,
without generating a flood of false alarms?

        Suerte,
                        _Vin 

 "Cryptography is like literacy in the Dark Ages. Infinitely potent, for good
and ill... yet basically an intellectual construct, an idea, which by its 
nature will resist efforts to restrict it to bureaucrats and others who deem
only themselves worthy of such Privilege."  
                  _A Thinking Man's Creed for Crypto  _vbm
                     
     *    Vin McLellan + The Privacy Guild + <vin () shore net>    *