Re: Assessment tools/Scanners

[gshipley () neohapsis com (Greg Shipley)]

On Mon, 11 Oct 1999, Stuart Staniford-Chen wrote:

> I'm not sure this is enough for a truly useful evaluation.  You also need 
>
> 3) knowledge about how likely a vulnerability is to actually get exploited.
>
> IDS's either don't keep up with fast networks at all, or just barely keep up
> under the right circumstances.  Every signature adds work that the IDS has to
> do, and makes it less likely that it will keep up.  So you probably don't
> want a product that will detect absolutely everything.  You want a product
> that will detect things most likely to be thrown at you.  And thus in a
> useful real-world evaluation, you should not be counting total
> vulnerabilities detected, you should be weighting the score by likelihood of
> vulnerabilities actually being exploited.

Ergh...I would argue that you want a product that CAN detect as much as
possible, but allows *YOU* to decide what it is going to look for.
Supposedly (and I'm attempting to confirm this) a few of the network-based
products can operate at 100Mbp speeds.  Hopefully I'll be able to publish
some solid numbers for everyone on this.  And the host-based ones
(obviously) can keep up at really high (100Mbps+) speeds.

> Unfortunately, "probability that attack technique Y will be used in a
> randomly chosen attack somewhere in the world today" is a little hard to
> measure :-).  And, like the list of all attacks known, it changes all the

Right, which is why I think the only way to objectively test network-based
systems is to include some level of signature verification.  For example,
we have identified a couple of vendors who claim to check for X when their
signature for X actually doesn't work right.  They of course corrected the
problem, but it proves this point: not all products work as advertised
(and shipped) and there is no verification body (or method) to prove this.

You see where I'm going with this?  :)

-Greg