Snort mailing list archives

Previous By Date Next
Previous By Thread Next

Re: Poor performance with Snort 2.9.4.6 under OpenBSD 5.3

From: Victor Roemer <vroemer () sourcefire com>
Date: Wed, 12 Jun 2013 12:15:42 -0400
First, I would recommend you tune your policy down. I believe for most
environments that this is not a
quick process; something to keep in mind.

Things to think about

preprocessor pop ...
preprocessor imap ...
preprocessor smtp ...

are each configured with

    b64_decode_depth 0
    qp_decode_depth 0
    bitenc_decode_depth 0
    uu_decode_depth 0

which permits unlimited decoding of the specified encoding. If your seeing
alot of email traffic,
you may do better to assign a fixed decoding depth.

Choosing an appropriate value that provides sufficient detection and
performance is where tuning gets tricky.

Similarly, if you don't need these preproc's you could also consider
disabling them.
(e.g., little traffic ever seen; or maybe you're rule-set doesn't benefit
with them enabled).


-------

preprocessor ssh ...

Without the appropriate preproc rules enabled, this preprocessor is not
likely to be doing anything for you.

You're also using the "autodetect" keyword in this preprocessor, which is a
performance
consideration as it will cause the preproc to look at traffic on all tcp
ports.


--------


preprocessor http_inspect_server:
...
    server_flow_depth 0

    client_flow_depth 0

Saving the best for last, HTTP Inspect will normally analyze the most
traffic of all the
application-level preprocs. The included options are easily the best place
to start tuning here.


--------

Also curious why you have 2 output-plugins configured, this is not helping
performance either.
Its recommended to use unified2, especially if you want to be fast.





Another thing to consider, and this should directly relate to Suricata
getting better numbers,
is load-balancing across multiple Snort processes.

Though I may be mistaken, Suricata should be doing across multiple threads,
rather than processes. (Anyone care to chime in?)


On Wed, Jun 12, 2013 at 3:08 AM, C. L. Martinez <carlopmart () gmail com>wrote:


On Fri, Jun 7, 2013 at 10:36 AM, C. L. Martinez <carlopmart () gmail com>
wrote:

On Thu, Jun 6, 2013 at 7:36 AM, C. L. Martinez <carlopmart () gmail com>

wrote:

On Wed, Jun 5, 2013 at 5:08 PM, Victor Roemer <vroemer () sourcefire com>

wrote:

Martinez, as Joel already mentioned, we'll want to see your Snort
configuration. Shutdown stats would also be useful, but perfmon data

would

be better; if those can be provided.

You mentioned that OpenBSD configured the network sysctl parameters

"on the

fly"; could you direct us to some documentation about this?

You also mentioned that Snort was listening on em3, however the startup
information in your email indicates that Snort is listening on em4,

could

you elaborate on this setup?


Regarding Suricata, I personally do not have any experience in

deploying or

configuring it. That said, are you using, relatively, the same
configurations? (e.g., any rules enabled, acquiring packets via

libpcap,

etc..)

Also, why are "tcp.reassembly_gap" and "tcp.invalid_checksum" relevant?



Ok, first of all sorry for this long post. I will try to answer to all
questions. First, my environment:

Host: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 8 GiB RAM
Snort: release 2.9.4.6 compiled against libpcap 1.3.0
Suricata: release 1.4.2 compiled against libpcap 1.3.0

Snort is listening in em4 interface and suricata in em5 (there was a
typo error previously).

Performance data using my actual snort.conf (with few rules and
without so_rules or IP based rules):

Packet Performance Monitor Config:
  ticks per usec  : 2405 ticks
  max packet time : 10000 usecs
  packet action   : fastpath-expensive-packets
  packet logging  : log
  debug-pkts      : disabled

Rule Performance Monitor Config:
  ticks per usec  : 2405 ticks
  max rule time   : 4096 usecs
  rule action     : suspend-expensive-rules
  rule threshold  : 5
  suspend timeout : 10 secs
  rule logging    : log
pcap DAQ configured to passive.
Acquiring network traffic from "em4".
Reload thread starting...
Reload thread started, thread 0x1ee8c530a500 (4050)
Decoding Ethernet

        --== Initialization Complete ==--

   ,,_     -*> Snort! <*-
  o"  )~   Version 2.9.4.6 GRE (Build 73)
   ''''    By Martin Roesch & The Snort Team:
http://www.snort.org/snort/snort-team
           Copyright (C) 1998-2013 Sourcefire, Inc., et al.
           Using libpcap version 1.3.0
           Using PCRE version: 8.31 2012-07-06
           Using ZLIB version: 1.2.3

           Rules Engine: SF_SNORT_DETECTION_ENGINE  Version 1.17

 <Build 18>

           Preprocessor Object: SF_DNP3  Version 1.1  <Build 1>
           Preprocessor Object: SF_MODBUS  Version 1.1  <Build 1>
           Preprocessor Object: SF_GTP  Version 1.1  <Build 1>
           Preprocessor Object: SF_REPUTATION  Version 1.1  <Build 1>
           Preprocessor Object: SF_SIP  Version 1.1  <Build 1>
           Preprocessor Object: SF_SDF  Version 1.1  <Build 1>
           Preprocessor Object: SF_DCERPC2  Version 1.0  <Build 3>
           Preprocessor Object: SF_SSLPP  Version 1.1  <Build 4>
           Preprocessor Object: SF_DNS  Version 1.1  <Build 4>
           Preprocessor Object: SF_SSH  Version 1.1  <Build 3>
           Preprocessor Object: SF_SMTP  Version 1.1  <Build 9>
           Preprocessor Object: SF_IMAP  Version 1.0  <Build 1>
           Preprocessor Object: SF_POP  Version 1.0  <Build 1>
           Preprocessor Object: SF_FTPTELNET  Version 1.2  <Build 13>
Commencing packet processing (pid=4050)


===============================================================================

Run time for packet processing was 368.552024 seconds
Snort processed 643495 packets.
Snort ran for 0 days 0 hours 6 minutes 8 seconds
   Pkts/min:       107249
   Pkts/sec:         1748


===============================================================================

Packet Performance Summary:
   max packet time       : 10000 usecs
   packet events         : 0
   avg pkt time          : 8.95128 usecs
Rule Performance Summary:
   max rule time         : 4096 usecs
   rule events           : 0
   avg rule time         : 1.96408 usecs


===============================================================================

Packet I/O Totals:
   Received:      2121798
   Analyzed:       643495 ( 30.328%)
    Dropped:       618918 ( 22.582%)
   Filtered:            0 (  0.000%)
Outstanding:      1478303 ( 69.672%)
   Injected:            0


===============================================================================


And here they are some statistics outputs:

top output:

load averages:  0.18,  0.18,  0.13
21 processes: 20 idle, 1 on processor
CPU0 states:  0.0% user,  0.0% nice,  0.0% system,  0.6% interrupt,

99.4% idle

CPU1 states:  1.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,

99.0% idle

CPU2 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,

 100% idle

CPU3 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,

 100% idle

Memory: Real: 506M/2942M act/tot Free: 5018M Cache: 2319M Swap: 0K/6142M

  PID USERNAME PRI NICE  SIZE   RES STATE     WAIT      TIME    CPU

COMMAND

 4050 root       4    0  417M  151M sleep/0   bpf       0:06  2.83%

snort

31522 root      10    0  356M  328M sleep/0   nanosle  35:09  0.73%

suricata


Status interfaces (systat ifstat):

    2 users    Load 0.17 0.17 0.13

IFACE            STATE  DESC
                     IPKTS   IBYTES    IERRS    OPKTS   OBYTES
OERRS    COLLS
em0              up
                         1       72        0        0       71
0        0
em1              up
                         0        0        0        0        0
0        0
em2              up
                         0        0        0        0        0
0        0
em3              up
                         1      334        0        2     1732
0        0
em4              up
                      6773  4771755        0        0        0
0        0
em5              up
                      6773  4771755        0        0        0
0        0

Interfaces configruation options:

em4:

flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>

mtu 1514
        lladdr 00:50:56:17:fc:cd
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet6 fe80::250:56ff:fe17:fccd%em4 prefixlen 64 scopeid 0x5
em5:

flags=8bc3<UP,BROADCAST,RUNNING,NOARP,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>

mtu 1514
        lladdr 00:50:56:18:79:51
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet6 fe80::250:56ff:fe18:7951%em5 prefixlen 64 scopeid 0x6

Checking mbuffers (systat mbufs):

    2 users    Load 0.18 0.17 0.13

IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
System                    0   256   361          59
                               2k   351         457
lo0
em0                            2k     7     4   256     7
em1                            2k     6     4   256     6
em2                            2k     4     4   256     4
em3                            2k     7     4   256     7
em4                            2k    69     4   256    69
em5                            2k    73     4   256    73

sysctl.conf options enabled:
kern.bufcachepercent=80
net.bpf.bufsize=65536
net.bpf.maxbufsize=2097152

Ok, now some Suricata stats:

5/6/2013 -- 14:29:09 - <Info> - stream "max-sessions": 262144
5/6/2013 -- 14:29:09 - <Info> - stream "prealloc-sessions": 32768
5/6/2013 -- 14:29:09 - <Info> - stream "memcap": 33554432
5/6/2013 -- 14:29:09 - <Info> - stream "midstream" session pickups:

disabled

5/6/2013 -- 14:29:09 - <Info> - stream "async-oneside": disabled
5/6/2013 -- 14:29:09 - <Info> - stream "checksum-validation": enabled
5/6/2013 -- 14:29:09 - <Info> - stream."inline": disabled
5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "memcap": 67108864
5/6/2013 -- 14:29:09 - <Info> - stream.reassembly "depth": 1048576
5/6/2013 -- 14:29:09 - <Info> - stream.reassembly

"toserver-chunk-size": 2560

5/6/2013 -- 14:29:09 - <Info> - stream.reassembly

"toclient-chunk-size": 2560

5/6/2013 -- 14:29:09 - <Info> - all 7 packet processing threads, 3
management threads initialized, engine started.
5/6/2013 -- 14:29:13 - <Info> - No packets with invalid checksum,
assuming checksum offloading is NOT used
6/6/2013 -- 06:27:20 - <Info> - Signal Received.  Stopping engine.
6/6/2013 -- 06:27:20 - <Info> - 0 new flows, 0 established flows were
timed out, 0 flows in closed state
6/6/2013 -- 06:27:21 - <Info> - time elapsed 57492.090s
6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Packets 7317731, bytes

5500227933

6/6/2013 -- 06:27:21 - <Info> - (RxPcapem51) Pcap Total:2974089715
Recv:2974089715 Drop:0 (0.0%).
6/6/2013 -- 06:27:21 - <Info> - AutoFP - Total flow handler queues - 6
6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 0  - pkts: 2754800
flows: 19921
6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 1  - pkts: 2158501
flows: 15267
6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 2  - pkts: 1276685
flows: 9932
6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 3  - pkts: 678835
flows: 5843
6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 4  - pkts: 358226
flows: 3109
6/6/2013 -- 06:27:21 - <Info> - AutoFP - Queue 5  - pkts: 127487
flows: 1488

-------------------------------------------------------------------
Date: 6/6/2013 -- 06:27:21 (uptime: 0d, 15h 58m 44s)
-------------------------------------------------------------------
Counter                   | TM Name                   | Value
-------------------------------------------------------------------
capture.kernel_packets    | RxPcapem51                | 2974088271
capture.kernel_drops      | RxPcapem51                | 0
capture.kernel_ifdrops    | RxPcapem51                | 0
decoder.pkts              | RxPcapem51                | 7317731
decoder.bytes             | RxPcapem51                | 5500227933
decoder.ipv4              | RxPcapem51                | 7317731
decoder.ipv6              | RxPcapem51                | 0
decoder.ethernet          | RxPcapem51                | 7317731
decoder.raw               | RxPcapem51                | 0
decoder.sll               | RxPcapem51                | 0
decoder.tcp               | RxPcapem51                | 7317660
decoder.udp               | RxPcapem51                | 0
decoder.sctp              | RxPcapem51                | 0
decoder.icmpv4            | RxPcapem51                | 71
decoder.icmpv6            | RxPcapem51                | 0
decoder.ppp               | RxPcapem51                | 0
decoder.pppoe             | RxPcapem51                | 0
decoder.gre               | RxPcapem51                | 0
decoder.vlan              | RxPcapem51                | 0
decoder.teredo            | RxPcapem51                | 0
decoder.ipv4_in_ipv6      | RxPcapem51                | 0
decoder.ipv6_in_ipv6      | RxPcapem51                | 0
decoder.avg_pkt_size      | RxPcapem51                | 752
decoder.max_pkt_size      | RxPcapem51                | 1506
defrag.ipv4.fragments     | RxPcapem51                | 0
defrag.ipv4.reassembled   | RxPcapem51                | 0
defrag.ipv4.timeouts      | RxPcapem51                | 0
defrag.ipv6.fragments     | RxPcapem51                | 0
defrag.ipv6.reassembled   | RxPcapem51                | 0
defrag.ipv6.timeouts      | RxPcapem51                | 0
defrag.max_frag_hits      | RxPcapem51                | 0
tcp.sessions              | Detect                    | 41460
tcp.ssn_memcap_drop       | Detect                    | 0
tcp.pseudo                | Detect                    | 5705
tcp.invalid_checksum      | Detect                    | 12
tcp.no_flow               | Detect                    | 0
tcp.reused_ssn            | Detect                    | 0
tcp.memuse                | Detect                    | 36175872
tcp.syn                   | Detect                    | 80827
tcp.synack                | Detect                    | 80035
tcp.rst                   | Detect                    | 33845
tcp.segment_memcap_drop   | Detect                    | 0
tcp.stream_depth_reached  | Detect                    | 148
tcp.reassembly_memuse     | Detect                    | 67770240
tcp.reassembly_gap        | Detect                    | 2222
detect.alert              | Detect                    | 29
flow_mgr.closed_pruned    | FlowManagerThread         | 49711
flow_mgr.new_pruned       | FlowManagerThread         | 3558
flow_mgr.est_pruned       | FlowManagerThread         | 1974
flow.memuse               | FlowManagerThread         | 3884096
flow.spare                | FlowManagerThread         | 10001
flow.emerg_mode_entered   | FlowManagerThread         | 0
flow.emerg_mode_over      | FlowManagerThread         | 0

As you can see, packets dropped counter shows 0% (well, in fact this
is not always the case. Suricata may lost between 1-2% of packets).
And this suricata sensor is configured to use IP based rules, when
snort not.
But there are two counters that can shows if some problems exists:
tcp.reassembly_gap and tcp.invalid_checksum. But as you can see, are
minimal according to Suricata docs:







https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Statistics





When I say: "OpenBSD tunes them "on the fly" according to network

load", I would say that OpenBSD 5.1 and later will dynamically adjust
the TCP send and receive window sizes. Please, refer to:

http://marc.info/?l=openbsd-misc&m=130804020930481&w=2
http://marc.info/?l=openbsd-misc&m=128904951710762&w=2
http://www.openbsd.org/faq/faq6.html#Tuning
http://quigon.bsws.de/papers/2013/AsiaBSDcon/cksums.pdf

Do you need to add more info or perform more tests?

And many thanks for your help.



Ok, more info. I have disabled SMP kernel version and changed by
uni-processor kernel (bsd) and now performance has grown: packets
dropped are between 5-20% now ...

Maybe all my problems are with system interrupts??




Interesting that this would be a problem, While I'm not well read on SMP, I
thought these kinds of
issues were only prevalent on single core hardware.







One question: it could be possible to use libpcap provided by OpenBSD

instead of install from www.tcpdump.org's site??

For example, I am trying to build daq against OpenBSD's libpcap and:

checking libnetfilter_queue/libnetfilter_queue.h usability... no
checking libnetfilter_queue/libnetfilter_queue.h presence... no
checking for libnetfilter_queue/libnetfilter_queue.h... no
checking for linux/netfilter.h... (cached) no
checking for pcap.h... (cached) yes
checking for pcap_lib_version... checking for pcap_lib_version in
-lpcap... (cached) yes
checking for libpcap version >= "1.0.0"... no

    ERROR!  Libpcap library version >= 1.0.0  not found.
    Get it from http://www.tcpdump.org

Suricata builds without problems using libpcap provided by OpenBSD ...

I have attached config.log.



This is a problem that we are already familiar with; the OpenBSD guys
maintain they're own customized
libpcap which is actually API compatible, but does not report as being
1.0.0 or greater.

Its certainly possible that the libpcap provided by them is faster, but I
really don't know.



Thanks.


------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!
Previous By Date Next
Previous By Thread Next

Current thread: