Re: Assessment tools/Scanners

[dugsong () monkey org (Dug Song)]

On Sat, 9 Oct 1999, Ron Gula wrote:

> Instead, we maintain logs of network attacks that we have collected
> with Dragon from places like DEFCON and SANS ID-Net. These logs may be
> trivially converted to TCPDUMP format for network replay.

trace-driven simulation is very useful, but there are a few problems in
using it alone for comparative evaluation.

1. good traffic is hard to find. DEFCON and SANS IDnet data tends to
   consist of ONLY attacks, with little or no "normal" background traffic.
   and without an authoritative list of attacks that actually exist in the
   trace, it's hard to compare IDS alert output.

2. even when the attacks are known, without a common attack taxonomy
   you end up comparing apples to oranges. ex. in the DARPA IDEVAL 
   (http://www.ll.mit.edu/IST/ideval/), 'mscan', 'nmap', 'SATAN',
   'NTInfoScan', etc. are listed as attacks - but these are actually tools
   that implement many discrete attacks, which are reported differently
   by different IDSs.

3. such testing is necessarily incomplete. by testing against a specific
   trace, you're demonstrating how well an IDS handles that particular
   dataset only. but what about an IDS's coverage of attack variants?
   or performance (and what metrics are you interested in, anyway)?

-d.

http://www.monkey.org/~dugsong/