Re: Recording slow scans

[Darren Reed <darrenr () reed wattle id au>]

In some email I received from Stephen P. Berry, sie wrote:
[...]
> > > What I want to do is twiddle libpcap [blah blah]
>
> > By the time you've done that you'll have wound up writing your
> > own NFR, or Bro, or argus or NNstat.
>
> You say that as if it was a bad thing.

How many times do people need to reinvent the wheel ?  Do you have
*that* much spare time ?  Maybe Marcus can shed some light on how
many man-days have been spent on tuning NFR's BPF implentation.
Maybe someone needs to donate some money to LBL so that BPF can
be "optimized" for performance with IDS's or just simply donate the code.

> that's entirely accurate.  I'm more interested in having a set of
> tools that can be incorporated into many different applications
> than in building the applications themselves.
[...]

For every tool you build into the pipeline of detection you have a
delay not much different to the one you mention below with packet
data moving up from the device.

What I suspect a few people would like to see is a "FWTK-like" set
of programs which Security Consultant Joe Bloggs can rip off and
build back-yard IDS's from to sell to unsuspecting companies.  I'm
not acusing you of that but I doubt I'd hear anyone complain (except
maybe those at NFR/ISS, etc).

IMHO, tcpdump output and storing that in a bunch of files, say monthly,
just doesn't cut it for real analysis of "what's happening".  For a start,
you can't make intelligent queries on data stored thus.  Paul was on the
right line talking about using Sybase with Linux.  You did finally arrive
at that point in your last paragraph but I'm not quite sure how a set of
tools will help you get there - unless you have a worthwhile design to
share.

[...]
> I think I'd rather see a sensor which could be distributed as a set
> of diffs against your favourite free UNIX source tree---that is,
> have the sensor smarts in the kernel.

Couple of problems here...
(1) potential loss of revenue for X companies which make IDS products;
(2) significant kernel bloat and subsequent requirements for machines;
(3) all IDS solutions are part-kernel, part-user programs;
(4) you have to convince the right people that it is worrthwhile (and
    they might (rightly) say "use NFR" or some such).

> This would allow you to
> grab the interesting bits of information from the datastream very
> close to the raw device---instead of waiting for the entire datastream
> to percolate up from the IP stack to luserland, to be sliced, diced
> and julienned there.

Which is exactly what BPF does.  Next.

[...]
> > Right. What you really need to be able to do is preload the
> > kind of analysis you want to do, so as much of it as possible
> > is done in realtime as the data comes in.  I haven't worked with
> > databases for a long time, but what it amounts to is pushing the
> > first stage of query optimization into the data gathering
> > loop -- then if the results come true, you run the rest of the
> > query.

But in the case of IDS, as we're talking here, that would result in
"open ended" queries which might be a new concept for database ppl(?) :)
By "open ended" I mean when you do your "select * where ip.src=internal"
it returns all those that it knows about now and then waits for the
next one - the query never really completes.

> Yes and no.  Certainly having a heuristic for evaluating how
> interesting a particular bit of traffic is while it is still hot off
> the wire is a good thing.

> But that's fundamentally just incident response,

Not necessarily "response".  Detection, sure, but that is the name of
the game (here).

> and I'm generally also interested in long-term trend
> analysis and suchlike as well---and I don't know of any way of getting
> away from looking at all the data while doing that.

"All" that data may not be important.