Re: BlackICE IDS

[jflowers () hiverworld com (John S Flowers)]

Robert,

[Warning: Serious opinion (based on experience) follows] 

This is not a Windows NT vs UNIX religious debate.  You'll note that in
my post I did not mention UNIX even once.  I mentioned a dedicated
Intrusion Detection Appliance and talked about "dedicated IDS"
technology.  While it is true that the NFR IDA runs on a stripped down
[OpenBSD] UNIX kernel, I didn't specifically mention UNIX as a superior
solution to Windows NT.

For example, I happen to admire the inexpensive Netscreen Firewall as
a viable and cost effective firewall solution at 10 or 100Mbit and 
it too is a dedicated appliance.  It's pretty darn fast as well, which
I attribute to it being a designed from the ground up to support only
the firewall functionality for which it was created.

Further, I resent the fact that you've put words in my mouth and called
this a religious debate between UNIX and Windows NT, which it is not. 
I'd say it's more of a debate between host based and network based IDS
technology, as the underlying comment that I left the reader with was
one related to resources for deploying a host based IDS across their
enterprise.

[Regarding your desktop configuration]
If you force processor affinity and run the IDS in a way that only the
developer of the technology could, you'll get better performance.  But
then, not many people can do that.  Also, the speed we're talking about
-- was that 148,800 FRAMES per second or 148,800 PACKETS per second? 
There's a difference.

I will maintain one point:  Any Network Security Appliance should
outperform any operating system + software solution.  Robert -- you
admitted that the mouse was jerky and the system was "still usable"
during the week long test that you performed.  So why bother running an
application that will cause user interface problems on the desktop? 
People have enough problems with their desktops being unstable [no
mention of an OS here] without adding the additional overhead of a 148k
frames/sec being handled under the hood.

Let me pose a logic problem to everyone:  Does it make more sense to
place one system on each network that can handle 100k packets/sec OR
place one service on each desktop, where the desktops are each handling
100k packets a second and experiencing possible usability problems?  My
support staff would shoot me in my sleep if I put them through the
headache of supporting Yet Another Desktop Application -- esp. one that
can be handled at the network level.

My suggestion was and -- for now -- is a Network IDS, preferably in
the form of a dedicated IDA.  I also believe that the IDS technology
arena is so new that we're still fumbling around in the dark, looking
for a light switch that will allow us to see the problem in front of
us.  If you don't believe me, please read the Greg Shipley IDS Review in
Network Computing on Nov. 15th, where he all but says, "IDS technology
is not ready for release."

http://www.nwc.com/1023/1023f1.html

I'd also like to point out that the NFR IDA received a 3.35 rating and
the Network ICE system received a 3.0 rating overall, and speed wasn't
the reason Black ICE received a lower grade -- feasibility was.

Further, do you care to share your traffic file so we can all start
talking around the same data?  I for one would like to see how other
technologies perform with the same set of traffic.  Surely you, as one
of the founders, don't mind sharing your data so we can all start
comparing apples to apples.  If anyone want's to look at a point of
comparison, my company offers some standard tcpdump formatted logs from
SANS NS'99s IDNET contest.  They can be found at
http://www.hiverworld.com/snortlogs -- although, there's a lot of really
strange stuff in these logs, so YMMV.

As everyone who knows me will attest, my opinion can be changed. It's
just that I'm having a really hard time with host based performance 
being sold as a neat little package that runs on the desktop in 
addition to all of the productivity applications that are also running
and are required for the desktop user to perform his/her job properly.

I have some software running here at Hiverworld that uses host based
agent software -- which communicates with a central IDS manager -- to
reconfigure the agents software rules in real-time.  This allows
us to prevent attacks without having the host analyze every little 
packet that hits the system.  It's a simpler solution, but it's still
a bit hard to make work properly, as the overhead is significant and
the desktop operating system sometimes experiences usability problems.

Robert Graham wrote:
> Hhhmm. The old WinNT vs. UNIX religious debate again.
>
> If the task isn't OS-bound, then the underlying OS just doesn't make a
> difference. (Just as the disk-drive subsystem doesn't make a bit of difference
> if you aren't saving to disk, or the video doesn't make a difference if you
> aren't writing to the screen). For an example, go to www.spec.org and check out
> the Intel SE440BX 450-MHz SPECint95 results that Intel runs on both WinNT and
> UnixWare; both platforms give identical results because SPECmark isn't OS-bound
> (the C runtime libraries make more of a difference than the OS).
>
> In my tests that captured+analyzed 148,800 frames/second, I ran on a
> dual-Celeron 450-MHz system. I used a utility from MS to force processor
> affinity: capture ran on one CPU, analysis on the other. The CPU doing the
> packet capture (which is OS-bound) took about about 50% utilization, whereas
> the CPU doing the IDS packet analysis took up about 90% of its CPU. The
> analysis does not interact with the operating system at all. Moving to a faster
> OS might help the packet capture, but it isn't the bottle neck.
>
> The test I ran used carefully chosen frames from real-world tracefiles with
> constantly varying IP addresses. I chose packets designed to test the major
> code paths in the sensor, such as TCP stream reassembly, connection-table
> management, etc. By matching up the counters, I confirmed that it captured and
> analyzed every one of the hundreds of millions of packets I generated. In all
> honesty, your mileage will vary widly. Some protocols take 10 times longer to
> decode than others; BlackICE is optimized for the most common ones, though.
>
> It's interesting to note that if you transmit 148,800 at a generic WinNT box,
> you'll notice that it takes roughly 50% of a 450-Mhz CPU to receive and discard
> the frames. You can run the tests yourself with a packet generator overriding
> the MAC address. This indicates to me that our capture driver adds very little
> overhead on top of the default WinNT behavior, and that capture is almost
> totally OS bound. We tried a number of different cards (3Com, Intel, DEC 21140,
> RealTek) and they all are pretty close in performance. But as I said, the
> bottleneck is analysis, which isn't OS-bound.
>
> Regards,
> Rob.
>
> --- John S Flowers <jflowers () hiverworld com> wrote:
> > ---
> > [This is a resend (from before 4PM PDT today).  The first message was
> > rejected.  My apologies if you receive two copies of this message.]
> >
> > First of all -- I haven't properly introduced myself to the list.  My
> > name is John S Flowers and I'm the founder and CTO (head geek) for a
> > small security company called Hiverworld.  We're located in Berkeley, CA
> > and we're the producers of a couple of pretty cool technologies that
> > perform real-time network security assurance and risk management
> > [imagine a version of ISS or CyberCop on steroids -- with 10x the number
> > of vulnerabilities and customizable options].
> >
> > We've been primarly privately held, with only a few Fortune 500 clients
> > funding our efforts, but we're beginning to take our technology public.
> > You can even search our vulnerability database on our website [we have
> > more than 1,000 public vulnerabilities listed].
> >
> > Anyway, enough about me.  On to the post.
> >
> > Am I reading this quote [below] correctly?  148,000 packets per second.
> > That can't be right.  We're talking about a Windows NT product that
> > requires the underlying hardware and software to be available enough
> > (processing-wise) for the IDS to perform properly.
> >
> > Most IDSs, even dedicated to the task of performing IDS, with a lot of
> > power and RAM, still can't perform this many operations.  I mean, if the
> > NFR IDA can't do 140k packets a second, how do you expect some Windows
> > system to perform?
> >
> > Oh, yeah.  You wanted advice.  "Unless you have 1,000 senior technical
> > security people on your staff to manage the software, buy a Network
> > IDS."  ;)
> >
> > P.S. Hey Ranum, speaking of cooking results.  Damn.  This *must* be a
> > misprint of the Network ICE claim of performance.
> >
> > --
> > John S Flowers                   <jflowers () hiverworld com>
> > Chief Technology Officer         http://www.hiverworld.com
> > Hiverworld, Inc.               Enterprise Network Security
> > Network Forensics, Intrusion Detection and Risk Assessment

-- 
John S Flowers                   <jflowers () hiverworld com>
Chief Technology Officer         http://www.hiverworld.com
Hiverworld, Inc.               Enterprise Network Security
Network Forensics, Intrusion Detection and Risk Assessment