Re: Testing IDS with tcpreplay

["Aaron Turner" <synfinatic () gmail com>]

No top post here...

On 2/23/06, Ivan Arce <ivan.arce () coresecurity com> wrote:
> Hi Aaron.
>
> See my comments inline as well ...
>
> Aaron Turner wrote:
> > Inline...
> >
> > On 2/20/06, Ivan Arce <ivan.arce () coresecurity com> wrote:
> > > Aaron Turner wrote:

[snip]

> If you are testing you IDS you'd like to know that it accurately detects
> an exploit for a given vulnerability not that it detects the packets
> that comprise a particular instance of the exploit's execution.

Well in both cases, you're only testing a particular instance of the
exploit.  If you want to try 500 different instances of a particular
exploit you have to run metasploit 500 times.  But when you test the
another IDS/IPS there is no guarantee that the next 500 times you try
it will look exactly like the first 500 times.  Using
tcpdump/tcpreplay you could capture those 500 tests and replay them
any number of times, thereby making sure that all devices see the same
thing which provides true comparative analysis.

> What I tried to point out is that with tcpreplay you end up testing
> different things than with Metasploit or similar tools. Using tcpreplay
> it good to build a valid and easily repeatable set of tests but it's
> important to know the difference with the real thing lest you'll get
> burned when a different exploit for the same bug or even the same
> exploit in a different operational environment hits your IDS.

As long as you understand the limits of the tools, and follow a
methodical approach, then you shouldn't get burned.  Don't get me
wrong, Metasploit is an awesome tool and belongs in everyone's
toolbox.  IMHO, it's much better at pen tests and testing targets
(where the goal is to "just get in") then doing comparative analysis
of IDS-like products.  Of course, replay tools NEED things tools
Metasploit to have something interesting to send in the first place.

[snip]

> > > > 3) Already have a library of pcap's (either from customers, the wild
> > > > or capturing traffic of real tools like Metasploit)
> > > Yeah, but that is an entirely different kind of testing. Replaying the
> > > packets captured from the execution of an exploit is not the same as
> > > executing the same exploit again.
> >
> > If you're testing a vulnerable application then I agree.  but if you
> > are testing an IDS/IPS,  then I would argue that it is for all intents
> > and purposes it's the same thing.  If you believe otherwise, then
> > please explain.
>
> Ok, I think I've touched on the point in my above comment but I will
> explain my view a bit more.
>
> I would argue that you are testing the IDS to figure out if it will be
> capable of detecting exploitation attempts against real applications in
> their real operational environment. If that is in fact the case, then
> you need to make sure that the exploit traffic you're using to test your
> IDS is likely the same that the IDS will see coming after the real
> applications. It follows then that if the exploit that generates the
> traffic will behave differently in the face of the real application then
> your captured-replayed traffic can't really assure that the IDS will
> work properly (unless you really understand how the exploit will behave
> and what traffic it will generate in any random execution).

I agree with this in one way:

If your goal is to test if the one or more IDS will detect the exploit
against a SPECIFIC target (ie, the hosts it will be deployed to
protect) then yes, replay tools suck.  But if you want to know if in
general one or more IDS's can detect an exploit against an arbitrary
instance of the target then replay makes sense.  The actual exploit
tool will work to of course, but introduces unknown variables; like
does the target behave differently the second or third time?  Of
course if you're not doing a regression or comparative test, then
using the actual tool provides everything you need.

> My point is that modern exploits do behave differently based on the
> runtime characteristics of the target system and application. I am not
> talking about shellcode obfuscation or NIDS evasion attempts but about
> exploit logic. For example if a given exploit needs to fill-up the
> process heap after/before triggering the bug or obtain heap layout data
> to change execution flow on the target app. then it will send more/less
> -or even different- traffic depending on the endpoint's application state.

Yep.  Which is why you should capture the same exploit being used
multiple times against multiple targets and then replay them. 
Otherwise you won't know if each IDS saw each of the different
exploitation methods.  Of course if you're not doing a comparative or
regression test, then you might not care (actually you might not care
if you are doing a comparative or regression test either, but I would
argue that you should :).

> > The point being you have to put forth the effort to automate
> > vmware/metasploit is a lot more work then just tcpreplay.  Not to
> > mention, if you want to add other tools other then metasploit now you
> > have more automation work to do.  tcpreplay is exploit/tool agnostic
> > and provides a simple and unified method of generating all kinds of
> > traffic- not just exploits.
>
>
> agreed and understood, I am just pointing out that you will accomplish
> different things depending on which test strategy you choose

Absolutely.  Different tools for different needs/strategies.
[snip]

> > Perhaps you're not aware, but tcpreplay fully emulates both client and
> > server state, from protocol setup, to the actual exploit and all the
> > way through getting a shell/whatever if that is what you captured.  An
>
>
> You mean at the TCP/IP stack level or does it emulate the application's
> state as well?

Since tcpreplay sends both C->S and S->C traffic *through* the IPS (or
in front of an IDS) it replays the externally visible state of the
TCP/IP stack, OS and application.  Obviously it doesn't track the
actual state of the target (by that I mean it doesn't track the stack,
heap or CPU registers) but if the attacker gets a shell and sends "ls
-l" to the target, the target will respond with a listing of the cwd
contents because that is what was originally captured.

The point being that the IDS/IPS doesn't actually track the state of
the target either, it only deduces the state by watching the traffic. 
Tcpreplay sends traffic which is completely indistinguishable from the
"real thing".

Of course as we've discussed, there are variables which might alter
how an exploit happens between one attempt and another, in which case
you should capture/replay multiple executions of Metasploit/etc.

> > IPS/IDS has absolutely no way of knowing if traffic was generated by
> > tcpreplay or not, because it is EXACTLY like the traffic that was
> > originally generated/captured.  If you want to run the same exploit
> > 500 times with different parameters with Metasploit, that's great.
>
> You can run the same exploit 500 times with exactly the same parameters
> and the generated traffic will not be the same. Some examples:
> - The Apache-SSLv2 exploit used by the Slapper worm
> - An exploit for the php_memory_limit bug
> - An exploit for MS License Logging Service (LLSRV)
>
> The reason for this is that they have logic to take different decisions
> or to use different data (pointers,etc) based on the heap state of the
> target application.

Right.  Which is why running Metasploit once isn't good enough to
answer the question,  "Does this IDS always detect Slapper?".   For
the same reason, you would have to capture/replay the same attack
multiple times.  But at least when you did so with a replay tool you'd
be able to answer the question, "Do these IDS/IPS's detect Slapper in
all the same cases?"

> > Just capture each of those 500 attacks and replay each of them against
> > 10 different IDS/IPS's.
> >
> > The point is that with tcpreplay, your test cases are 100% repeatable.
>
> Yes, but that does not assure 100% code coverage of the exploit you used
> to generate the captured traffic.

Right now I'm not aware of any testing methodology that assures 100%
code coverage for an exploit.

> As I said, I see the value of using tcpreplay but the implications need
> to be clearly understood so you know what you're testing.

Agreed, but that's true of any tool.

--
Aaron Turner
http://synfin.net/

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
------------------------------------------------------------------------