Re: Testing IDS with tcpreplay

[Stefano Zanero <zanero () elet polimi it>]

Aaron Turner wrote:

> > > 1) Trying to do comparative analysis and you want to make sure each
> > > device sees exactly the same thing
> >
> > Hmm, why is that harder to accomplish with Metasploit than with tcpreplay?
>
> Because metasploit, other tools and exploits incorporate PRNGs and
> other methods of altering the attack so that it isn't exactly the same
> each time.   

Everything depends on defining what you want to test.

If you want to test the ability of detecting that exploit, you also want
to detect the ability of detecting its variations.

Therefore, you want to generate the attack in as many flavors and random
numbers as possible, and note down how many times each device catches
it. Then, the fact that these variations are "not the same" should get
less and less important as the number goes up.

If this still seems "unfair" to you, either you have very little faith
in IDS algorithms you are testing, or perhaps this is not what you
really want to test.

> That makes "each device sees exactly the same thing"
> really difficult.   

Yes, but there is no reason to require that. It's a misconception of
"scientific repeatability".

> Again, less complex (no 2nd box and vmware to maintain/automate) 

This is surely true. However, how do you GENERATE that replayed stream ?

And once you have an architecture in place for generating it... why not
use it also for testing ?

>  Also what about attacks that Metasploit
> doesn't have?  

You run them yourself ?

> What if you want background traffic?

You generate it ?

Your objections can be drawn exactly in the same way if you want to use
tcpreplay.

Unless you are suggesting to use some well-known-as-broken repositories
of data such as the DARPA datasets. You aren't proposing that, are you ?

> If you're testing a vulnerable application then I agree.  but if you
> are testing an IDS/IPS,  then I would argue that it is for all intents
> and purposes it's the same thing.  If you believe otherwise, then
> please explain.

Some context-aware systems for instance would behave differently if an
exploit is executed onto a vulnerable application or onto a
non-vulnerable application. Anomaly detectors could not be able to catch
the attack itself, but may be very well able to see the consequences,
etcetera.

> Say you have two IPS's you want to test.  You an send an "attack" with
> Metasploit against the first one and it detects it.  You run it again
> against the second one and it doesn't.  

Consider this.

You send a tcp replayed stream against the first one, and then agains
the second one. The first one catches an attack, the second one doesn't.

Does this mean the first one is better ? No, it does not mean absolutely
anything.

The core of the problem is in WHAT YOU CALL TESTING, not in what you use
to throw packets at your IDS probe...

Regards,
Stefano

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------