Home page logo
/

snort logo Snort mailing list archives

Re: Anyones doomsday machine running low on IDS analyst tears?
From: Will Metcalf <william.metcalf () gmail com>
Date: Thu, 7 Oct 2010 05:19:12 -0500

Steve,

Thanks for the additional info.  It was more of a heads up for folks
switching to a VRT snort/2.9.0 conf to review their existing
rule-sets. If people have custom rules updating to this version of the
conf can have a significantly negative performance impact for some
sigs.  Of course if you don't update to include other portions of the
VRT/snort-2.9.0 config like extended_response_inspection you don't
have access to file_data so VRT sigs that rely on this won't fire.

Regards,

Will


On Wed, Oct 6, 2010 at 10:01 PM, Steven Sturges
<steve.sturges () sourcefire com> wrote:
The max_pattern_len is a memory option that is used in conjunction
with ac-split (introduced in 2.8.6).  In terms of the pattern matcher
itself, it gets near AC-FULL performance at the AC-BNFA memory.
Pretty significant when there are 10k+ rules in your configuration.
Performance-wise, reducing the size of the pattern matcher is two-fold.

-- Use of the faster AC algorithm
-- Fewer CPU cache misses during pattern matching phase

The whole idea of the fast pattern matcher is to use the most unique
patterns to select a set of "most likely" rules for further evaluation.

One could argue that of the pattern "User-Agent|3a| Mozilla/3.0
(compatible|3b| Indy Library)", the real unique part starts with
"Mozilla", so the pattern itself could be shortened.  Every HTTP
request has a User-Agent header these days, so that really isn't buying
you all that much in terms of uniqueness of the pattern itself.  And
when it comes to evaluation of the content, that's an extra 12
characters that need to be compared via boyer-moore.

In the specific rule example, if "User-Agent:" is a required part
of the pattern, using an offset of 13, length 20 would get you
"Mozilla/3.0 (compati" -- eg: content:"User-Agent|3a| Mozilla/3.0
(compatible|3b| Indy Library)"; fast_pattern:13,20;
Pretty unique & short.  Therefore less memory in pattern matcher --
that can now be used for Stream reassembly, gzip decompression, etc.

Will's solution in the follow-on post of fast_pattern:30,20 also works
and gives you the tail of the above pattern.

Or, split the two contents:

content:"User-Agent|3A| "; http_header; content:"Mozilla/3.0
(compatible|3b| Indy Library)"; distance:0; fast_pattern;

Or, increase the max pattern length in the default config.

Rule writers know the most unique part of the content patterns
they are using, especially as they relate to patterns in other
rules, and the parameters to fast_pattern give them the tools
that are needed.

Snort is highly configurable and users have so many options to work
with here to suit their specific ruleset, sensor, available memory,
network speed, etc.

Or, you can simply revert to using the slower AC-BNFA configuration
of yore.

Cheers.
-steve

From: *Will Metcalf* <william.metcalf () gmail com>
Date: Wed, Oct 6, 2010 at 3:10 PM
To: Emerging Threats Signatures <emerging-sigs () emergingthreats net>
Cc: Snort Users <snort-users () lists sourceforge net>


I kid... I kid.. But seriously ran into something sort of interesting
playing snort conf options.  It seems that the default pm in snort has
changed the the default pattern matcher to be ac-split via...

# Configure the detection engine  See the Snort Manual, Configuring
Snort - Includes - Config
config detection: search-method ac-split search-optimize max-pattern-len 20

If you are using your own custom rule sets/ET rules this means that if
you previously had an override for fast_pattern's default cut-off of
20 via offset and length options via fast_pattern:<offset>,<length>;
or you expected the entire unique pattern specified via fast_pattern,
You are now limited to only 20 bytes for fast_pattern even if you set
offset/length. So for example let us consider the following ET
signature optimized for 2.8.6 using the default pm in 2.8.6 of
ac-bnfa.

alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET TROJAN
Delf Checkin via HTTP (8)"; flow:established,to_server;
content:"POST"; http_method; content:".php"; http_uri; nocase;
content:"User-Agent|3a| Mozilla/3.0 (compatible|3b| Indy Library)";
http_header; fast_pattern; content:"name="; http_client_body; depth:5;
classtype:trojan-activity;
reference:url,doc.emergingthreats.net/2008268;
reference:url,
www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/VIRUS/TROJAN_Delf;
sid:2008268; rev:5;)

 Fast pattern matcher: HTTP Header content
 Fast pattern set: yes
 Fast pattern only: no
 Negated: no
 Pattern offset,length: none
 Pattern truncated: no
 Original pattern
   "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
 Final pattern
   "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"

timestamp: 1286391049
Rule Profile Statistics (all rules)
==========================================================
  Num      SID GID Rev     Checks   Matches    Alerts
Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
  ===      === === ===     ======   =======    ======
=========  =========  ========= ============
    1  2008268   1   5         18         4         4
        55        3.1        3.8          2.9

timestamp: 1286391054
Rule Profile Statistics (all rules)
==========================================================
  Num      SID GID Rev     Checks   Matches    Alerts
Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
  ===      === === ===     ======   =======    ======
=========  =========  ========= ============
    1  2008268   1   5         18         4         4
         52        2.9        3.8          2.7

timestamp: 1286391058
Rule Profile Statistics (all rules)
==========================================================
  Num      SID GID Rev     Checks   Matches    Alerts
Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
  ===      === === ===     ======   =======    ======
=========  =========  ========= ============
    1  2008268   1   5         18         4         4
       54        3.0        3.9          2.8


Ok this is what I expect. I have a long match that should be fairly unique.

Lets modify the pm to be the one included in the VRT version of the
default snort.conf and/or the 2.9.0 snort.conf.  We now get
"User-Agent:|20|Mozilla/" as the match added to fast_pattern.  Uh
oh... This isn't a very unique pattern is it, this will cause all
packets that contain a firefox UA to be at least partially evaluated.

1:2008268
 Fast pattern matcher: HTTP Header content
 Fast pattern set: yes
 Fast pattern only: no
 Negated: no
 Pattern offset,length: none
 Pattern truncated: 50 to 20 bytes
 Original pattern
   "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
 Final pattern
   "User-Agent:|20|Mozilla/"

timestamp: 1286390771
Rule Profile Statistics (all rules)
==========================================================
  Num      SID GID Rev     Checks   Matches    Alerts
Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
  ===      === === ===     ======   =======    ======
=========  =========  ========= ============
    1  2008268   1   5      17381         4         4
10212        0.6        4.7          0.6

timestamp: 1286390775
Rule Profile Statistics (all rules)
==========================================================
  Num      SID GID Rev     Checks   Matches    Alerts
Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
  ===      === === ===     ======   =======    ======
=========  =========  ========= ============
    1  2008268   1   5      17381         4         4
9516        0.5        4.7          0.5

timestamp: 1286390779
Rule Profile Statistics (all rules)
==========================================================
  Num      SID GID Rev     Checks   Matches    Alerts
Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
  ===      === === ===     ======   =======    ======
=========  =========  ========= ============
    1  2008268   1   5      17381         4         4
9347        0.5        4.5          0.5

timestamp: 1286390784
Rule Profile Statistics (all rules)
==========================================================
  Num      SID GID Rev     Checks   Matches    Alerts
Microsecs  Avg/Check  Avg/Match Avg/Nonmatch
  ===      === === ===     ======   =======    ======
=========  =========  ========= ============
    1  2008268   1   5      17381         4         4
9266        0.5        4.1          0.5

Evaluating the same pcap we now spend 20x more ticks evaluating the
same rule.  I doubt that specifying a fast pattern offset and length
will allow us to override the max-pattern-len 20 but lets give it a
shot by specifying 0,50 as the offset and length.

snaplen = 65535
1:2008268
 Fast pattern matcher: HTTP Header content
 Fast pattern set: yes
 Fast pattern only: no
 Negated: no
 Pattern <offset,length>: 0,50
   "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
 Pattern truncated: 50 to 20 bytes
 Original pattern
   "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)"
 Final pattern
   "User-Agent:|20|Mozilla/"

No dice.. So I guess the take away here is that if you are moving to a
VRT snort.conf or a 2.9.0 ruleset and you are running custom rules I
would pay real close attention to debug-print-fast-pattern output.  We
are going through the poor performers now and making modifications
where appropriate for ET rules, just thought folks might want to know
;-)...

Regards,

Will


------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
Spend less time writing and  rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users


  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]
AlienVault