RE: IDS testing methodologies

[<Raj_Dhingra () NAI com>]

Henrik,

First of all, IDS testing is a complex subject. I have mentioned the
names of a few independent test organizations that I am aware of below. 

As an IDS/IPS vendor, we (Network Associates) have invested
significantly in testing and benchmarking our products. It requires
resources (people with a security background that understand intrusions,
vulnerabilities, protocol analysis etc. and understand how to set up the
test equipment properly). It requires equipment like WebAvalanche to
generate the traffic ( the equipment in some cases will cost you more
than your IDS/IPS pilot purchase!).

When evaluating an IDS, it is important to identify the various criteria
that are critical for you to meet your organizational objectives. Some
of these criteria could be technological (detection accuracy, false
positives, evasion resistance, attack detection at high throughput
rates, range of IDS/IPS response actions etc), some could be operational
(ease of installation, usability, update capability, MTBF etc.), some
could be business related ( cost of solution, technical support, track
record of vendor providing timely security advisories/response, future
product roadmap etc.)

Your question below focuses on the technology testing. 

Some factors to consider for IDS Testing ( This focuses only on
detection testing not on installation, operations and maintenance
impact)

- What attacks to use for testing? ( You can use actual traffic from
your network by connecting your IDS in passive mode or  do a traffic
capture with sniffer and replay or  obtain attack tools/scripts for lab
test)
- Traffic mix (If doing a lab test) - what types of  background traffic
do you have in your test enviroment to simulate real-world conditions.
HTTP, DNS, FTP, SMTP etc. This will affect what equipment you need to
generate such traffic. There are organizations such as caida
(www.caida.org) that publish information about typical traffic mix.
- Additional Traffic Parameters( If doing a lab test): packet size (eg:
HTTP request and response sizes etc.) 
- Load testing ( eg: increasing background traffic from 100 Mbps to
multi-gbps, increasing connections/sec etc)

I would suggest that you explore the following resources from
independent test organizations. Each of them have written about their
IDS testing methodologies that expand in detail about some of the
factors I mentioned above and much more. 

> > > The NSS Group  https://www.nss.co.uk . Bob Walder of the NSS Group
has done multiple iterations of IDS and Gigabit IDS testing over the
years. The NSS Group is working on an IPS test right now.

> > > Neohapsis http://osec.neohapsis.com/. Neohapsis has also done
several iterations of IDS testing over the years and have published
their Open Security Evaluation Criteria (OSEC). Greg Shipley, the CTO of
Neohapsis, drives these criteria and also contributes to Network
Computing Magazine on product testing including IDS testing. Full
details about their methodology is at the link above.

> > > Miercom is part of Network World's Global Test Alliance and did a
gigabit IDS test for Network World in November 2002. They wrote about
their test methodology in
http://www.nwfusion.com/reviews/2002/1104revhow.html


IMHO:  
1) There is no 'ultimate test methodology'. This is a dynamic and
evolving area for now. You could learn from any or all of the above. 

2) IDS testing can be resource and capital intensive if you want to do a
thorough job of setting up an environment for lab test 

3) Several independent test organizations have experts that specialize
in this and have published their results. So, you should be clear on why
you want to do your own test ie. Are there any criteria that are not
covered in these published tests? Or are you evaluating how the product
performs in your environment? Perhaps it may be useful for you to talk
to one of these organizations for advice and/or to influence their
criteria.

Regards

Raj Dhingra



-----Original Message-----
From: Henrik Falkenthros, direktoer [mailto:hef () bridicum dk] 
Sent: Tuesday, December 30, 2003 11:43 AM
To: focus-ids () securityfocus com
Subject: IDS testing methodologies


Hi List !

I'm trying to find out ways of testing different IDS systems; is there a
'recommended'/best practise methodology for testing Network based IDS
(NIDS) ? Any information - papers, tools, links and own experience are
much appreciated,,, 8-)

cheers, Henrik Falkenthros


------------------------------------------------------------------------
---
------------------------------------------------------------------------
---


---------------------------------------------------------------------------
---------------------------------------------------------------------------