RE: Intrusion Prevention requirements document

["Tony Haywood" <thaywood () karalon com>]

Matt,

" I strongly believe that replay tools are NOT an effective way to test an
IPS:"

That's quite a bold statement to make.  I agree that they are not a panacea
but not effective?  If that was the case then why do tools such TCPReply,
Tomahawk and even the Metaspolit project exist other than to replay in a
controlled manner, live or pre-captured sessions of an exploit to its
natural conclusion?  And why are these very tools used by the majority of
the security vendors to augment the design and validation of signatures not
to mention the testing labs in their relevant reports?  

As the NSS group was highlighted in the original response, here is a link
specific to their previous IPS Group test where these test tools are
discussed in relation to their use in the group test:

http://www.nss.co.uk/ips/edition3/appendix_b.htm

Using a single tool as the only means of vendor comparison is of course over
simplifying the problem but should be done so in the context of the end user
expectations.  Replay tools augment existing testing methodologies and can
reduce the overhead of lab design, hardware and knowledge requirements.  

Its worth remembering that not everyone working in this field understands
how to use all of the tools (or the operating systems they run on), build
vulnerable targets and fix the exploit code which usually requires some
tinkering to prevent the average script kiddie from using it. Then of course
there are the problems with some aspects/exploits with widely accepted tools
such as Metaspolit (generally an excellent quality product but we are aware
of some exploits that have issues) and the problem of finding the time to do
all this.

Again to highlight this point I would begin to look at the myriad of
external security validation organisations that exist today and in
particular, the tools in which they use.

If all that is required is a means to determine if the device being tested
is functioning, can detect industry recognised exploits and act in
accordance with security guideline then replay tools are an ideal solution.

If rate detection, variants and evasion technique detection are required
then these same tools still provide a valid service.  In the case of Traffic
IQ Pro and IDS Informer, both solutions include variants for this very
reason. In Traffic IQ Pro's case these extend to variants created using the
Metaspolit Project and when combined with various layer 2,3 and 4
modifications the applications support, do provide a high degree of
validation.

I haven't forgotten about evasion in relation to fragmentation, obfuscation
etc, instead I built a layer 3 solution that does all this without an IP
stack and called it the Evasion Gateway.  Used in conjunction with testing
tools such as Traffic IQ Pro you have a very powerful testing solution that
doesn't have any requirement for additional "live" hardware or is limited by
the usual single IP constraints of many of the testing tools you have
mentioned.

Volume is always a concern.  There are open source tools out there that
purport to be able to generate large volumes of traffic but unless a
hardware solution is used, rate is determined in a large case by the
hardware being used, operating system, resource utilisation, intermediate
devices and the quality of the data streams (as a start).  

If we take a look at Tomahawk for instance, it has the capability for load
generation but only based on packet captures so that fails in your criteria
immediately.

Other hardware only solutions do however have the capability for the live
generation of multiple traffic streams but again care and understanding
needs to be used in their configuration.  A device generating GB's of random
packets is not going to stress a stateful device as much as one that can
transmit stateful sessions and even some of the hardware devices warrant
detailed investigation.  Our own research has found worrying concerns with
at least one high profile vendor that generates packets with invalid chksum
values for TCP packets.  An IPS should easily detect this and drop the
packets but not for the reasons the test was devised.  Again this comes back
to a thorough understanding of the issues and tools being used which is not
always the case.

You raised a question about pcaps and the amount of time that they have been
around.  Many threads on this forum have asked where these pcaps can be
harvested from for use in applications such as tcpreplay or tomahawk that
unlike Traffic IQ Pro do not have their own library.  The conclusion has
typically been that these are proprietary and therefore not generally
available.  Even the ones that are supposed to be supplied with open source
tools such as Tomahawk have been notoriously difficult to get hold of in the
past. 

To the best of my knowledge the only applications that do contain an
extensive library of pre-defined, verified and focused captures of actual
attacks are IDS Informer and Traffic IQ Pro.  Of course we still need to ask
why, if these pcaps have been around for so long, do so many still evade a
number of IDS/IPS solutions on the market today?

As for zero day, well there really is very little that can be done here
other than constant vigilance and a thorough understanding of what your
security device is telling you.  Unless of course your system employs
anomaly detection capabilities in which case the probability if correctly
identifying zero day exploits greatly increases.

Without plenty of time, extensive knowledge, continual research, training
and the reliance on third party solutions/companies this is very difficult
for many people which is why there are testing companies out there such as
the Tolly Group and NSS that use tools such as Traffic IQ Pro in their own
labs both with the supplied libraries and augmented with their own captures.

And one area that hasn't been raised is the cost/ROI associated with many of
the "big tools" for IPS validation.  This is an area that replay tools of
this ilk clearly show their worth.

I openly admit that I am coming at this from a vendor perspective but I
think that this is warranted given the nature of the original mail.  My
intention is not to promote the use of my products as I know that this is
not the correct forum to do so, but it is to try and provide an unbiased as
possible response to questions that have been raised in my peer group.

Tony

-----Original Message-----
From: FinAckSyn [mailto:finacksyn () yahoo co uk] 
Sent: 08 November 2005 11:08
To: Arun Vishwanathan; vendortrebuchet () comcast net; thaywood () karalon com;
focus-ids () securityfocus com
Cc: Tony Haywood; pen-test () securityfocus com
Subject: RE: Intrusion Prevention requirements document

Hi VT,

I strongly believe that replay tools are NOT an
effective way to test an IPS:

1)  Replay tools are an unfair way to compare vendors.
 There is no measure as to the speed at which an IPS
vendor responded to the original vulnerability.  These
.pcap files have been around for months or even years,
giving plenty of time for vendors to catch up and
write signatures to stop them.  It's all very well
having an IPS that responds favorably to a replay
tool, but if certain signatures took days, weeks or
even months to write, then this is not a fair way to
compare device A with device B.
It can be difficult to get hold of, but try and get
the 'time to respond' stats from your proposed IPS
vendor.  

2)  Replay tools do not test variants.  Although the
pcap content may reflect a specific exploit, what
about all the other exploits that abuse the same
vulnerability?  For example, the 50+ odd variants of
Blaster, Slammer or Code Red?  These test tools
usually include pcaps for one particular variant only.
 A good freeware tool to test variations is
Metasploit, or if you have spare cash, Canvas is a
worthwhile investment.

3)  Replay tools do not test ability of a device to
withstand volume.  When worm/virus outbreaks happen,
you don't just get a single packet in .pcap fashion
that comes in, trips a signature, and gets blocked. 
You usually get several million of them.  This is
where it is important to test any rate-based features
of the device.  Also make sure these rate-based
features don't block valid traffic.  Plenty of
freeware tools available - Apache Benchmark, nmap,
Nessus, hping2 and a SYN Flood tool called Juno. 
What's more, these tools do not rely on .pcaps, so are
a lot more real world.  Be warned that replaying an
identical pcap several million times is not an
accurate way to rate-test, seeming all L2/L3
information is identical in each packet (eg no change
in SEQ).  Devices that are good at DDOS tend to be
just as good at withstanding sudden, large propagation
of worms.  

4)  Replay tools do not test IPS avoidance.  Well, you
may get vendor-supplied pcaps like
blaster_with_fragments.pcap, but there are so many
other ways to evade an IPS, and can you be sure that
the vendor supplied .pcap is a reflection of real
world traffic?  Get hold of fragroute, tcpsic, plus
nessus has some good evasion options.

5) Replay tools do not test zero-day protection.  This
is the fun bit.  How do you test a network IPS
protects you against the shape of things to come?
If you have a research team that can generate
signatures to protect you quickly, then great.  But
there's still a window of opportunity during which
your security can be breached.
If an IPS is firing back events such as Sasser.W32/A
blocked, or Code.Red.B.W32/Z, then chances are, it's
not an anomaly based system that will give you much in
the way of zero-day protection.  But if it's firing
events like 'CIFS Field Too Long' or 'HTTP Header
Contains Illegal Characters', then this is indicative
of the machine having good anomaly/zero-day procetion,
rather than specific signatures for specific
pre-historic viral events...  ;)

IPS testing is a big task, and needs big tools - to do
things properly, take a look at the NSS testing suite,
for example -  

http://www.nss.co.uk/utm/appendix_a/appendix_a.htm

In conclusion (I hear sighs of relief...):

* Replay tools cannot be solely relied on to test an
effective IPS 
* Think about what you want your network IPS to do -
content-based checks are important, but equally as
important are access control and rate-based ability. 
* A network IPS will never provide 100% perimeter
protection.  Always invest in extra security layers,
especially host-based, to ensure that anything the IPS
lets through does not cause problems
* If you buy an IPS on the merits of tcpreplay
results, you risk being hit with a zero-day threat or
DoS condition, and losing your job.
* Treat any vendor that promotes testing with a replay
tool with caution (I learnt the hard way..)

Hope this helps,

Regards,

Matt


--- Arun Vishwanathan
<arun.vishwanathan () nevisnetworks com> wrote:

> Hi VT,
>
> I have used IDSInformer myself for testing and it is
> a very good
> product. There is a similar free tool (but lacks
> certain features)
> called Tomahawk which was released by Tippingpoint
> some time back.
> (http://tomahawk.sourceforge.net/)
>
> The working of these tools is very simple. You have
> to assign two
> interfaces. The tools consider one interface as
> "client" and other
> interface as the "server". The PCAP can be easily
> split into two parts,
> client traffic and server traffic. Consider the
> following simple packet
> sequence (A and B are IP addresses). 
>        
> 1. A -> B SYN (client)
> 2. B -> A SYN-ACK (server)
> 3. A -> B ACK     (client)  
>
> Packet 1 is first sent out on client interface. The
> packet is expected
> to arrive on interface 2 within a certain timeout.
> On receipt of packet
> 1, packet 2 is sent out on interface 2. Then packet
> 3 is sent out on
> interface 1 on receipt of packet 2 and so on. They
> make the IDS believe
> that it is seeing a real traffic situation. 
>
> In informer, you can change the MAC, IPs, Sport,
> Dport of the packets.
> In tomahawk you can only change the IPs at present
> but if you want to
> you can easily modify the code as its very simple.
> There is no need to
> configure any networks on the interfaces etc. Infact
> the IPs, MACs can
> be spoofed because it really doesn't matter. 
>
> Tomahawk has one limitation that it cannot test a
> Layer 3 device because
> it lacks support for specifying the source gateway
> MAC and Destination
> gateway MAC. It can test only Layer 2 devices.
> Informer can be used in
> both L2 and L3 situations. 
>
> In my opinion, both tools are great. I have used and
> am using both tools
> extensively. Informer also has an evaluation
> version. You can download
> it and try for yourself. For both the tools very
> little configuration is
> required. 
>
> Hope I was able to clear some of your doubts. 
>
> Regards,
> Arun
>
> -----Original Message-----
> From: vendortrebuchet () comcast net
> [mailto:vendortrebuchet () comcast net] 
> Sent: Sunday, November 06, 2005 6:11 AM
> To: thaywood () karalon com;
> focus-ids () securityfocus com
> Cc: Tony Haywood; pen-test () securityfocus com
> Subject: RE: Intrusion Prevention requirements
> document
>
> This sounds like a very viable solution that will
> allow for testing.   I
> assume that it replays both the stimulus and
> response of any
> conversation and does not "fingerprint" the packets
> at any layer with
> the host OS TCP/IP stack (e.g. change of window
> size, TTL, etc)?  Does
> the product automatically adapt to replay source and
> destination traffic
> based upon reading a libpcap file or do you have to
> configure the
> networks per card?
>
> Has anyone else used this or a similar product in
> their testing or other
> security product tests?  What issues did you
> encounter?
>
> Thanks for the feedback,
> -VT
>
>
> > One of the ways that you could test safely is by
> using something like
> > Traffic IQ Pro or a similar product. It is a
> stateful traffic replay
> tool
> > and can be used to test any inline or packet
> monitoring device. 
> > The product uses two network cards and so the
> library of over 700
> normal and
> > threat traffic files can be replayed statefully
> without the need to
> connect
> > to a live target system. This allows for live
> production systems to be
> > testing for the correct configuration really
> quickly and easily. 
> > I have been involved in working in this area for a
> number of years now
> and
> > my previous company was Blade Software where I
> developed IDS Informer
> and
> > Firewall Informer to provide similar testing
> capabilities.  
> > Information on Traffic IQ Pro is available below
> should you want to
> take a
> > look. 
http://www.karalon.com/Karalon/TrafficIQ/TrafficIQ.htm
> > Working with testing labs and a number of security
> and networking
> vendors
> > has enabled Traffic IQ Pro to be a really useful
> tool for anyone who
> wants
> > to check the configuration of their firewalls,
> IPS, IDS, routers,
> switches
> > etc and see how those devices perform under
> different scenarios. 
> > Tony
> >
> > Tony Haywood
> > www.karalon.com 
> >
> >
> > -----Original Message-----
> > From: vendortrebuchet () comcast net
> [mailto:vendortrebuchet () comcast net]
>
> > Sent: 29 October 2005 20:40
> > To: focus-ids () securityfocus com
> > Subject: Re: Intrusion Prevention requirements
> document
> > Another question for everyone,
> > When you brought in each vendor for evaluation,
> did you configure a
> test
> > network for them or did you use your production
> network?  My 1st
> concern is
> > keeping my job :o)  If I test in production, I
> could impact production
> > traffic.  If I don't test in production, how can I
> best ensure that I
> won't
> > have problems with custom applictions, older IP
> stacks which could be
> an
> > issue if RFC compliance checks are done, etc.  
> > The vendor answer is always, "don't turn on
> blocking and just
> monitor."  Is
> > that a reality?   I'd like some testimonials to
> this and some real
> life
> > instances of what has been done from unbiased
> sources.
> > Thanks,
> >
> > VT
> >
> >
> > > All,
> > >
> > > I work on a team that manages signature and
> behavioral based
> intrusion 
> > > detection systems today.  We have been tasked
> with reviewing IPS (or
>
> > > whatever vendor name acronym you prefer) in '06.
>  Our normal process
=== message truncated ===



                
___________________________________________________________ 
To help you stay safe and secure online, we've developed the all new Yahoo!
Security Centre. http://uk.security.yahoo.com





------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------