Re: threshold -- is it really deprecated?

[Patrick Mullen <pmullen () sourcefire com>]

Hello, all!

I'd like to explain a bit of why we deprecated threshold and replaced
it with detection_filter and event_filter.  It's actually mostly what
Russ said -- to split detection capabilities from reporting
capabilities.  Another reason was to fix a "deployment issue" that was
causing reporting thresholds to not work as expected.  I'll explain
this "issue" below, and I'll mention something that we use in the VRT
to make this mostly a non-issue.  There is a TL;DR version at the end
if you're impatient.  But I think the prose is important and necessary
to explain some of the TL;DR.

First, the easy one -- detection_filter.  This is directly analogous
to "threshold: type threshold."  It tells snort to not alert (and not
block, if run in blocking mode) until a particular event is seen
excessively.  An example of where we would use this type of filter
would be when we are alerting on traffic that is normal unless it's
seen more than an acceptable level.  An example of this would be sid
17429, where we track HTTP 404 responses and alert only if we get 100
in 30 seconds going to the same host.  404s happen all the time, but
if there are that many in such a short amount of time, someone is
fishing (with an 'f'; not phishing).

event_filter was created because it is purely a reporting-side
function.  This is an important distinction.  For detection_filter on
a drop rule, snort allows traffic to pass through unchanged until that
detection_filter threshold is reached and then it kills packets.
event_filter does nothing of the sort.  It just limits how much an
alert will spam your console.  Having this information in the rule in
a single instance of snort ends up being pretty much exactly the same
as what we have with the two types of filters separated.  But where it
gets funky is when you have (potentially multiple) instances of snort
and a central management console.

If each sensor only reports an event every X times due to filtering,
then the management console only reports an event every Y times, you
will only see the event at most every X*Y times due to double
filtering.  If a rule previously used "threshold: type both" (delay
before alerting then limit reporting), the problem was even worse.
Due to snort delaying its first report, then limiting the frequency of
its alerting, your management console may not ever alert at all
because its own alerting threshold may not get triggered.  This means
you would be dropping traffic without ever seeing anything on your
central console.

Here's the fix --

Split the functionality that restricts the initial alerting from the
reporting mechanism.  By having detection_filter in the rule, we
operate as we always wanted to: allowing traffic up to a certain point
then blocking it when we've determined it's beyond the norm.  Once the
detection_filter is tripped, the rule will alert every single time it
matches until the threshold is no longer met.  To limit the spam of
events that is then created, there is event_filter that you can put
into your snort instance if you're running just a single snort or you
can do your filtering on your management console.  Doing it this way
ensures we never again have a scenario where we are double-filtering
alerts, dropping packets, and not seeing any notification of doing so.

Now, for the "why it all works out in the end" --

Russ is also correct that it's rather a "tool chain" issue that we
don't deliver an event_filter.conf that would possibly make this
discussion not necessary.  Change is scary, and the hassle of having
to edit two files is not lost on me, but it really is the more
flexible, more powerful, and less confusing way to do things.  This is
true if for no other reason than if you want to put an event_filter on
a sid, you no longer need to search for that sid in your rules file.
You just put it anywhere in event_filter.conf and be done with it.
And now your local copy of the rule is not modified from the official
version so if we update the rule's detection (or detection_filter) you
don't need to worry about merging the new version of the rule with
your updated logging filter.

But in reality, it's actually even easier than this.

Distributing an event_filter.conf has been put on a fairly low
priority because snort supports global thresholds.  Analyzing the rule
set before we made this change, we found that the predominant
"threshold: type limit" was to squelch malware alerts to once every
minute or once every few minutes.  By putting a global threshold
within snort of one alert per minute per sid on a host on all rules
achieves this goal.

TL;DR version:
1) Easier to manage and conceptualize
2) Avoids "filtering multipliers"
3) Allows users to modify alerting characteristics without modifying
the rule detection
4) Global reporting thresholds makes this mostly a non-issue


I hope this helps clear some of the confusion.


Thanks,

~Patrick

------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel

Please visit http://blog.snort.org for the latest news about Snort!