Re: Recording slow scans

["Stephen P. Berry" <spb () incyte com>]

-----BEGIN PGP SIGNED MESSAGE-----


Darren Reed <darrenr () reed wattle id au>:

> How many times do people need to reinvent the wheel ? 

Until there's a GPL'd wheel out there that I happen to like.  Because
sometimes all I want is the wheel, and most vendors are car salesmen.


[arguments about freely-available open-source code deleted]


> Paul was on the
> right line talking about using Sybase with Linux.  You did finally arrive
> at that point in your last paragraph but I'm not quite sure how a set of
> tools will help you get there - unless you have a worthwhile design to
> share.

Actually, I suspect the overall design looks similar in both cases.
Someone asked if anyone had experience with getting IDS data into
a database and analysing it.  In responding I observed that I generally
see two bottlenecks:  one in my first-order filtering;  and one during
the analysis in the database itself.  The former bottleneck is
clearly a fuction of the fact that I'm using tcpdump as part of
my filtering.  Using NFR instead is certainly one alternative.  Building
a new set of tools or improving the existing tools are two other
alternatives.

If there is no interest in discussing the latter two scenarios, that's
fine.  The second bottleneck appears to be common to both schemes (NFR
and !NFR)---performance of the database itself.




> > I'm generally also interested in long-term trend
> > analysis and suchlike as well---and I don't know of any way of getting
> > away from looking at all the data while doing that.

> "All" that data may not be important.

`Important' is a leading word, but I think I know what you mean.

If I was a betting man, I'd lay money that all the data -isn't- `important'.
And if you know of a way I can determine whether or not data is important 
without looking at it, I'd certainly like to hear about it.

By the time data reaches the database, I've already checked it for
most of the sorts of bad things I can think to look for (or at least
the ones I care about)---i.e., it's already been compared to known
intrusion/attack/probe signatures.  So what's left to look for by
the time the packets reach the database?

The obvious answer (or at least the one which seems obvious to me) is
to determine a baseline and then look for anomalies.  What remains
an open question, however, is how to best set about doing this.

I'm open to suggestions.  Generally speaking I look at a number of
different things, including:

        -Simple statistical analysis.  Keep track of the average
         level of traffic to and from particular hosts, and set a flag
         if it varies one or two sigma from this mean.  Keep
         track of how traffic is distributed across the hours of the
         day, and across days of the week, then flag on odd spikes or
         lulls.  That sort of thing.
        -Repeated patterns of access.  I.e., source host tries to connect
         to everything from /etc/services on one host;  source host tries
         one or two ports on every host in a subnet;  u.s.w.
        -A list of twinks.  A list of hosts (or networks) from which
         suspicious traffic has originated before.  Keep track of everything
         coming from them.  Correlate against the above.


Of course figuring out what's field and what's foreground is one
of the recurrent problems of intrusion detection.  That's one of the
reasons I asked if anyone was experimenting with using inductive learning
algorithms to accomplish this---it seems like such a beast might be able
to handle some of the patternmatching that human operators have
traditionally done.  How many IDSen are configured to flag on
repetitive frag size and offset pairs, or notice when it sees
a sequence number a few too many times, or whathaveyou?











- -Steve


-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQCVAwUBNiKScCrw2ePTkM9BAQEJpQP/f9LWocztHRiJel0F04U2uGVW2z0DjyIk
nJn1MigtLUBXow1UNGmWXEHMbGePiBoR5fThn6fqlVNLwjZIVaUmb5yLsZwqsYAt
dwQczZzigZIRn3GeuRjyjQ2+QE+Md+ncGj1CitcEaClaDBGQ3GbuADdrQOV+W4zZ
OxUi9EnhXMo=
=6qhI
-----END PGP SIGNATURE-----