Re: Intrusion Detection Evaluation Datasets

[Stefano Zanero <zanero () elet polimi it>]

> Actually, I think it is: the majority of unique NIDSs that I am
> familiar with were built to use the KDD Cup '99 dataset. I pray none
> of those systems are actually used in production anywhere.

Let's hope not.

On the other hand:

> it, only a handful of signature based network intrusion detectors were
> ever built.

There are also non-signature based systems that do not look only at the
headers. Most notably PAYL and ALAD, and even my own work.

> really haven't been any major changes to signature based detection in

Well, I'm pretty sure the Bro guys would raise an exception on that ;-)

> the past decade (just thousands of tweaks). Most anomaly or machine
> learning based detectors will only work with structured data, so they
> limit themselves to the header portions of the packets or connection
> records.

Once again, while this is mostly true, it's not completely true, see above.

> should it require unique training data for a given network, is it
> feasible that such training data will ever be available?

This is feasible only if the system can be trained on non-labelled (and
not clean) data.

> I see a lot of people saying (correctly) that advanced (non-signature
> based) NIDS can't be researched until we have good evaluation
> datasets, and I see a lot of people ignoring them and doing it anyway.
> Is anyone (else) actually working on fixing the data problem?

Actually, the problem cannot be easily fixed, as Stuart Staniford so
well describes in a subsequent message.

-- 
Cordiali saluti,
Stefano Zanero

Politecnico di Milano - Dip. Elettronica e Informazione
Via Ponzio, 34/5 I-20133 Milano - ITALY
Tel.    +39 02 2399-4017
Fax.    +39 02 2399-3411
E-mail: zanero () elet polimi it
Web:    http://home.dei.polimi.it/zanero/