There is another issue to consider with IPS testing that you do not
see with IDS testing. While you can verify the detection capability of
an IDS/IPS using these tools if the captures you use show a complete
exploit (and not just the packets that one vendor thinks are
interesting), it is much more difficult to verify the protection
responses of an IPS using such tools. Detecting is a simple matter of
verifying that a reasonable event triggers when you replay the
capture. However, IPS devices are active devices and work by modifying
the network traffic that flow through them in response to undesired
events. This produces a variety of challenges.
First, once the IPS responds, the remainder of the packets replayed
are likely no longer an accurate reflection of what would have
happened with live traffic. For example, if the IPS drops packets,
does the capture show retransmissions at the key points? If not, how
do you know that naturally occuring retransmissions wouldn't get
forwarded? Does the tool compensate for this by synthesizing
retransmissions on "dropped" packets?
What if the IPS inserts TCP RST packets ahead of the undesired packet
instead of dropping it? Does the tool account for this? How does the
tool know whether the RST would have been effective? Does the tool
understand the protocol stacks it is simulating to be able to answer
that question?
What if the IPS doesn't respond by dropping packets? What if it
rewrites the attack out of the packet before forwarding it in some
cases (because simply dropping or resetting an SMTP connection tends
to make matters worse rather than better for example)? How does the
tool know whether this rewriting would be effective?
Ultimately, you get into a situation of role-reversal. How do you test
these tools? Well you exercise them in "real world" scenarios to see
how they behave. That is, you place them in a test bed with an IPS.
You launch "attacks" and see what happens. If the test fails, you look
to see if the tool failed or the IPS failed. Very rapidly you get into
the situation where the IPS is used as the "test tool" for your test
tool. That is, the IPS measures and heavily influences "proper" tool
behavior. Tomahawk works well at testing Tippingpoint's products
because that is how it was tested. It is horrible at testing ISS
products (the vendor I happen to work for) because Tippingpoint has no
vested interest in investing time and energy to make sure that ISS,
Netscreen, McAfee, etc (I apologize to all those I left out) are
adequately measured by their tools. I do not see how placing the tool
in the public domain solves this problem.
So who is going to invest $1,000,000 in IPS equipment from various
vendors to properly test this tool and make sure it gives every vendor
fair representation? Who gets to decide what is fair? The vendors? The
public domain developers? Do you want to be one of those developers? I
can just smell the defamation lawsuits from whichever vendors feel
they are not being fairly represented by your code changes...
FYI, I work for a vendor. We have a tool very similar to Tomahawk. It
works well at testing our products. However, other vendor's products
do not stand up as well to our testing :)
The only reliable means of testing IPS product effectiveness I have
discovered so far is live fire testing. Setup an isolated LAN. Place
vulnerable systems on one side of the IPS and launch attacks from the
other side. If the vulnerable system is compromised, the IPS failed. I
strongly recommend modifying all of the exploits to only DoS the
victim so that nothing more than a reboot is ever necessary to prepare
for the next test.
Paul
On Sat, 6 Nov 2004 13:16:02 -0800, ADT <synfinatic_at_gmail.com> wrote:
> (thread is getting long, so just going to snip the whole thing,
> hopefully you kept a local copy)
>
> Hey Greg/Marty,
>
> I don't think anyone would argue that tcpreplay or tomahawk are
> written for performance
> testing of IDS or IPS. I'm sure some people do that, but both have
> rather limited use in that regards (you want to generate background
> traffic using *your* network's traffic). What tcpreplay and tomahawk
> do rather well is provide the means to safely reproduce malicious
> traffic for testing detection capabilities.
>
> Unlike "live tests", tcpreplay/tomahawk don't require people to
> distribute working exploit code
> or attack an actual host which due to the nature of exploits will
> likely have to be "fixed" in some
> manner. Unlike exploit code, there is no risk that a pcap will also
> re-format your harddrive or
> require you to install and configure a wide variety of operating
> systems and applications to
> attack.
>
> Of course, unlike a "live test" there is some trust involved that the
> pcap contains packets which
> are relevant for the test you are running. Wether or not this
> precludes using either tool for being
> used by someone evaluating an IDS/IPS probably depends on how much
> they trust the pcaps.
> For those people who don't want to trust pcaps and don't have the
> means to get a library of working exploits, I'm sure Blade will be
> more then happy to sell you IDS Informer (of course, now you have to
> trust Blade, so you're just shifting your trust).
>
> Of course if you already have a repository of valid pcaps (maybe
> something the OSVDB guys could do?) with known attacks, then using
> these tools probably make a lot of sense for certain kinds of tests.
>
> Aaron, the tcpreplay guy
>
> --
> http://synfin.net/
>
>
>
> --------------------------------------------------------------------------
> Test Your IDS
>
> Is your IDS deployed correctly?
> Find out quickly and easily by testing it with real-world attacks from
> CORE IMPACT.
> Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
> to learn more.
> --------------------------------------------------------------------------
>
>
--------------------------------------------------------------------------
Test Your IDS
Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
--------------------------------------------------------------------------
Received on Nov 09 2004
Intro
