Re: kernel implementations

[Martin Roesch <roesch () hiverworld com>]

Archive: http://msgs.securepoint.com/ids
FAQ: http://www.ticm.com/kb/faq/idsfaq.html
IDS: http://www-rnks.informatik.tu-cottbus.de/~sobirey/ids.html
HELP: Having problems... email questions to ids-owner () uow edu au
NOTE: Remove this section from reply msgs otherwise the msg will bounce.
SPAM: DO NOT send unsolicted mail to this list.
UNSUBSCRIBE: email "unsubscribe ids" to majordomo () uow edu au
-----------------------------------------------------------------------------
Hi Everyone,

Since John brought it up, let me interject for a moment and talk a bit about
"Target-based IDS".  Target-based IDS lets us retain the manageability of the
NIDS-type distributed sensor model while approaching the accuracy of
host-based systems (in terms of network based attacks, anyway).  There are
several advantages to this approach over traditional NIDS:  

a) accuracy (which leads to)
b) reduction in false alarms
c) speed

TIDS is, by design, more accurate than NIDS.  The detection engine and rules
structure of a TIDS is dynamically generated each time it initializes or
updates, laying out the set of servers and their associated services into the
data structures and functional tests required to examine traffic on the
network for signs of intrusion.  Once this picture of the network is built we
can do a number of cool things including target based IP defragmentation &
stream reassembly (so long, and thanks for all the fish Mr Ptacek & Mr
Newsham), OS/application specific
rules, rejection (or direct shunting to log) of traffic that's heading for
hosts that are not alive on the network (and/or heightened analysis of that
traffic like a network sniffing "honeypot").  

The design of this system is predicated upon the existence of a continuously
running and highly accurate network vulnerability scanner.  Fortunately,
Hiverworld just happens to make one of those.  You have to have this because
the TIDS must be updated seamlessly and automatically every time the network
changes and because if the scanner isn't accurate the TIDS won't be either.

There are two *huge* advantages that this approach buys us.

Reduction of false alarms.  There was just a (fairly worthless) article in
eWeek this morning
(http://www.zdnet.com/eweek/stories/general/0,11011,2606343,00.html) that
discussed how people were turning off many of the functions of their NIDS due
to constant false alarming.  This is a familiar story from the IDS designer's
perspective, and one of the things we strive to improve upon.  Unfortunately,
the network is a harsh mistress when it comes to trying to optimize a NIDS for
optimal rules loading and false alarm reduction. :)  The traditional method
for false alarm "tuning" is to set the NIDS up for a day or two then analyze
its output for intrusive data (i.e. attacks) and squelch out the noisy rules. 
This isn't exactly a great method for real world use.  

The obvious example of why it isn't a great method is where a NIDS is
configured to ignore things that are still potentially vulnerabilities but
aren't anyplace on the network, then later a machine with this vulnerability
is added to the network.  One of two things happens: 

a) The false alarm problem is still there so it's left as an "acceptable risk"
and left out of the system

b) Nobody thinks/remembers to update the IDS rules set and it's left
unmonitored

In this scenario, the user is lucky if there are some secondary effects of an
attack that set off the NIDS by accident or are noticeable in some other way,
but if those don't show up the attack is completely missed. 

The other big advantage of TIDS is speed.  The detection engine is much faster
because it can select the proper rules to test more rapidly and because fewer
rules need to be checked for a given server/vulnerability.  Current IDSs
(well, Snort at least, probably some others as well) need to *find* the right
set of rules to be tested (based on protocol/IP/port information) before it
can actually begin testing them.  A TIDS doesn't necessarily have to do this
since it knows how large the network is before hand and which hosts are alive
so it can set itself up appropriately to only lookup the servers that exist,
arranging those servers into a rapid lookup data structure.  At the same time,
a given entry for a server in the detection engine only needs to contain tests
relevant to that particular OS/architecture and application (and we can get as
specific as versions and patch levels) that the packet is hitting.  Leveraging
the information collected by the vulnerability scanner in this way reduces the
number of tests that need to be performed for a given packet dramatically,
which means that the detection engine (and the rest of the program) is that
much faster.

Here's another example.  If your network is going to be running both IIS and
Apache web servers, a NIDS would need to load and monitor all traffic for both
types of attacks, potentially traversing the complete set of IIS and Apache
vulnerabilities for every web server access.  Since many NIDS do content
checking for these services (which is an expensive operation) these rules are
processed slowly.  In a TIDS we can select the proper server rules directly
and only test the things that (for example) Apache 1.3.9 and the particular
version of MiniVend (or whatever) are vulnerable to.  Non-attacks can be
logged as anomalous data, while attacks can be recorded and reported with high
fidelity.  Additionally, a TIDS will automatically recognize when a web server
is running on a "weird" port (say 111) and configure itself appropriately
since it knows not only the ports that are open for a given host on the
network, but also which applications are running on those ports.  Try *that*
with a standard NIDS. :)

Tie this to a super fast packet collection/filtering system (HwBPF) and an
excellent reporting system (Hiverworld Ansible) and I'd say (but maybe I'm
biased) you've got a pretty solid mechanism for monitoring your network for
attacks.

   -Marty


John S Flowers wrote:
> Archive: http://msgs.securepoint.com/ids
> FAQ: http://www.ticm.com/kb/faq/idsfaq.html
> IDS: http://www-rnks.informatik.tu-cottbus.de/~sobirey/ids.html
> HELP: Having problems... email questions to ids-owner () uow edu au
> NOTE: Remove this section from reply msgs otherwise the msg will bounce.
> SPAM: DO NOT send unsolicted mail to this list.
> UNSUBSCRIBE: email "unsubscribe ids" to majordomo () uow edu au
> -----------------------------------------------------------------------------
> Robert,
>
> Our tests show that we can easily handle 32,000-40,000 packets a second,
> but we're sending realistic traffic to the IDS and not 60 byte packets.
> I often wonder if we can talk Anzen into doing another real world study
> of IDS speed differences between NFR, Network ICE, RealSecure, CyberCop
> Monitor, etc.  [Allen, you out there?]
>
> There's another performance boost that we gain because we're using what
> Marty Roesch [after he started work here] started calling a "Target
> based IDS" or TIDS [tm]. ;)
>
> Basically, we already have a fairly successful *distributed*
> vulnerability scanner & risk management product [now called Swarm,
> formerly called ARMS] and we're able to leverage the Swarm information
> by using the Ansible [centralized management & reporting console -- see
> the white paper on our site] to gather & share data with the IDS.
>
> What does this mean in practical terms?  The most notable difference
> between our TIDS and normal NIDS is that we can maintain a detailed
> network picture of the customer network, including host operating
> systems and vulnerabilities.  Next, we use this information to our
> advantage by _not processing_ any packets that are destined to a host
> that either a) doesn't exist or b) doesn't have any open ports.
>
> In practice, this helps us get past many of the insertion, evasion & DoS
> attacks that Ptacek & Newsham proposed as we have an a priori
> understanding of the network before the IDS is even turned on.  It also
> allows us to simply log most of the information and only attempt to use
> the detection engine when we're looking at a packet that is destined for
> a valid host/port combination.
>
> Unfortunately, if you're looking for a "network flight recorder" or
> similar, we're not the right choice.  We don't capture every single
> packet on the network.  If you're looking for a high speed network IDS
> that logs only valid attacks (and not false positives -- when the target
> system isn't vulnerable to an attack) we're absolutely going to be the
> right choice for your environment.  Heck, you may even want to have both
> systems for various reasons.
>
> As yet another side note, I'll be speaking in Vegas next week
> [http://www.defcon.org/html/defcon-8-pre.html#John Flowers ] on this
> very subject and would encourage anyone who wants more information to
> attend my presentation & Robert Grahams as well [he's speaking a bit
> before me at Defcon].  Just be sure to ask Robert a lot of hard
> questions. ;)
>
> Robert Graham wrote:
> > --- John S Flowers <jflowers () hiverworld com> wrote:
> > > First and foremost, I may have mispoke when mentioning that we pass
> > > pointers around that point to the NICs memory.  What we're actually
> > > doing is copying the packet from the NIC to a kernel mapped memory
> > > segment that is exposed to userland.  This allows us to read all of the
> > > packet information from a shared memory location either by the kernel or
> > > by the userland application.
> >
> > I was wondering about that. We (Network ICE) looked at directly accessing the
> > card's buffers and quickly realized that PCI overhead would stall the CPU. In
> > other words, its a lot faster having the card DMA-copy the data to memory than
> > for the CPU to access the data on the card. Since the DMA transfers are
> > unattended by the CPU, this has little impact on processing.
> >
> > In other words, "zero-copy" these days really refers to the work the CPU has to
> > do. Most OSes are based upon this zero-copy architecture because it is
> > extremely efficient.
> >
> > > NIC -> copy by kernel to memory map <- available to userland
> >
> > We've been doing this kernel<->userland shuffle for the past two years. As far
> > as I can tell, this is the only way to really do things efficiently. You
> > absolutely want the IDS running in userland, but you must tweak packet capture
> > performance in the kernel/drivers.
> >
> > > The speeds we're getting are tough to measure without having everyone
> > > one this list scream that we're either "cooking our results" or similar,
> > > but I will say this:  on a P3/700 (coppermine) using the SysKonnect
> > > SK-9843 (recommended to us by Theo) with 128M of RAM we're seeing ~300
> > > Mbps [sustained].
> >
> > The only people who would claim you are cooking the results are people that
> > haven't had experience with Dragon or BlackICE Sentry. In any event, the real
> > question is how many packets/second you are getting, since that is the real
> > issue that kills most products. The max packet rate on Ethernet is 148,800
> > packets/second using 60-byte packets, but equates to only about 65-mbps. Most
> > products rated at 100-mbps cannot handle even 20,000-packets/second.
> >
> > =====
> > Robert Graham  http://www.robertgraham.com/pubs
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Get Yahoo! Mail - Free email you can access from anywhere!
> > http://mail.yahoo.com/
>
> --
> John S Flowers                   <jflowers () hiverworld com>
> Core R&D                         http://www.hiverworld.com
> Hiverworld, Inc.       Continuous Adaptive Risk Management

-- 
Martin Roesch                      <roesch () hiverworld com>
Core R&D                         http://www.hiverworld.com
Hiverworld, Inc.       Continuous Adaptive Risk Management