Re: Unusual Snort performance stats

[Matt Watchinski <mwatchinski () sourcefire com>]

Couple things to look at that might help track this down.

1. Outstanding means that packets never got out of the ethernet card before
they got dropped.  IE pcap didn't get to them before they disappeared.  One
thing to check here is the type of ethernet card you are using and if you
are running the latest drivers for that card.  I normally see best
performance out of Intel cards.  Obviously the stats are borked for some
reason since they are > 100%, but its something to consider.

2. Feb 22 03:30:12 snortsensor1 snort[4567]:     OTHER: 72674000   (18.281%)

This stats means that some percentage of your traffic contains protocols
that snort doesn't do anything with.  Tracking these down and add BPF's to
ignore them could improve performance.

3. Are you using CPU affinity to lock the snort process to a specific CPU?
If not this is something to try.  If snort bounces to another CPU then the
cache line is reset and performance can suffer.

4. Upgrade to snort 2.8.5.3, just in case.

Cheers,
-matt

On Mon, Feb 22, 2010 at 8:37 AM, Willst Mail <willstmail () gmail com> wrote:

> Hello,
> We are seeing some strange statistics in Snort stats.  When I look at
> syslog following a restart of Snort, this is what I see for
> performance:
>
> Feb 22 03:30:12 snortsensor1 snort[4567]: Run time prior to being
> shutdown was 86389.32948 seconds
> Feb 22 03:30:12 snortsensor1 snort[4567]:
>
> ===============================================================================
> Feb 22 03:30:12 snortsensor1 snort[4567]: Packet Wire Totals:
> Feb 22 03:30:12 snortsensor1 snort[4567]:    Received:    201246607
> Feb 22 03:30:12 snortsensor1 snort[4567]:    Analyzed:    396896876
> (197.219%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:     Dropped:      2798169
> (1.390%)
> Feb 22 03:30:12 snortsensor1 snort[4567]: Outstanding:
> 18446744073511103178 (9166238551048.516%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:
>
> ===============================================================================
> Feb 22 03:30:12 snortsensor1 snort[4567]: Breakdown by protocol
> (includes rebuilt packets):
> Feb 22 03:30:12 snortsensor1 snort[4567]:       ETH: 397535427  (100.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   ETHdisc: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:      VLAN: 238577055  (60.014%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:      IPV6: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   IP6 EXT: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   IP6opts: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   IP6disc: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:       IP4: 397110659  (99.893%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   IP4disc: 18109940   (4.556%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:     TCP 6: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:     UDP 6: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:     ICMP6: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   ICMP-IP: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:       TCP: 270400570  (68.019%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:       UDP: 34726192   (8.735%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:      ICMP: 1380875    (0.347%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   TCPdisc: 3          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   UDPdisc: 292        (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   ICMPdis: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:      FRAG: 126408     (0.032%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:    FRAG 6: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:       ARP: 95294      (0.024%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:     EAPOL: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:   ETHLOOP: 25887      (0.007%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:       IPX: 0          (0.000%)
> Feb 22 03:30:12 snortsensor1 snort[4567]:     OTHER: 72674000   (18.281%)
> Feb 22 03:30:13 snortsensor1 snort[4567]:   DISCARD: 18110235   (4.556%)
> Feb 22 03:30:13 snortsensor1 snort[4567]: InvChkSum: 2915       (0.001%)
> Feb 22 03:30:13 snortsensor1 snort[4567]:    S5 G 1: 219175     (0.055%)
> Feb 22 03:30:13 snortsensor1 snort[4567]:    S5 G 2: 365456     (0.092%)
> Feb 22 03:30:13 snortsensor1 snort[4567]:     Total: 397535427
> Feb 22 03:30:13 snortsensor1 snort[4567]:
>
> ===============================================================================
> Feb 22 03:30:13 snortsensor1 snort[4567]: Action Stats:
> Feb 22 03:30:13 snortsensor1 snort[4567]: ALERTS: 911
> Feb 22 03:30:13 snortsensor1 snort[4567]: LOGGED: 911
> Feb 22 03:30:13 snortsensor1 snort[4567]: PASSED: 0
> Feb 22 03:30:13 snortsensor1 snort[4567]:
>
> ===============================================================================
>
> Somehow we are analyzing 197% of packets, and we have a remarkable
> number of outstanding packets (I'm not even sure what "outstanding"
> packets are).  We do a restart of Snort every 24 hours, and these
> stats are pretty typical.  We are consistently around 193-198%
> analyzed, and some ridiculous number for outstanding (1844.... never
> the same).  This is v2.8.5.1 (build 114), configured with
> "--enable-sourcefire --enable-targetbased --enable-perfprofiling."
> Snort is running as a daemon and is listening on a single interface.
> That interface is receiving the transmit and receive lines from
> multiple ISP links (eventually we will be adding additional sensors so
> each only monitors a single ISP link).  We haven't done much rule
> tuning yet, and the only output directive is for unified2.  This
> machine has 4gb RAM and 8 cores (dual quad-core 3.0gHz Xeons?) with
> Snort typically using around 40% of one core.
>
> Any ideas why our performance statistics could be so unusual?
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users<https://lists.sourceforge.net/lists/listinfo/snort-users%0ASnort-users>list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users



-- 
Matthew Watchinski
Sr. Director Vulnerability Research Team (VRT)
Sourcefire, Inc.
Office: 410-423-1928
http://vrt-sourcefire.blogspot.com && http://www.snort.org/vrt/

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users