Gigabit and Snort

["Gross, Mark" <mgross () microstrategy com>]

Hello List,

 

I am having a few problems with losing packets at high speeds. First a
bit about the hardware. Assume that all connections are fiber.

                      

Cisco 65xx <---------> Cisco 65xx (failover with 4 fiber trunks)

        |                              |  <-- Sniffing Interfaces

Snort Sensor            Snort Sensor  

        |                              |

        |         Database       |

        |               |              |    <-- Management Network

        ------ Cisco 65xx ------ 

 

Each sensor is a Dell 2650 , 2 x 2.8 Xeon, 4GB Ram, 720GB Internal
Storage, 2 x Intel 1000 PCI-X Gigabit NICs.

The Database is a Dell 2650, 2 x 2.8 Xeon, 12GB Ram, 720GB Internal
Storage, Powervault Array with 1.7TB Storage using the PERC4 Raid
Controller, and 1 x Intel 1000 PCI-X Gigabit Nic.

      

OS is RedHat Enterprise Linux AS.

 

Users passing through the sensors (via 14 VLANS) are about 800. All
traffic and users MUST pass the sensors, even for internal
communications.

 

That's the basic setup.

 

So my problem is that the sensors are dropping ungodly amounts of
traffic at very high speeds (see below).  I have read some of the other
articles about the buffers and all, and on one box I set them and it
showed minimal improvement.  So I am kind of lost as to a cause.  Here
are the stats of one of the sensors. There arn't actually 4 CPUs ,
that's the Hyperthreading crap.. DoH. Snort is hanging right around 50
Procent CPU time and Memory is good.  There is no logging on the sensors
and everything is going over the management network to the MySQL
database. When I do a tcpdump, snort on the interface you can see the
packet loss (see below). Also included are the Interface stats.  The
card is not maxed either.  I plan on setting up Barnyard very soon, but
I can't imagine this would create such a packet loss. The snort.conf is
a straight out of the box with the home_net as ANY.  Otherwise just the
MySQL output plugin is configured.

 

Thanks in advance.

 

Mark

 

 

------------------------------------------------------------------------
------------------------------------------

 

 

CPU/Memory Statistics

 

21:40:31  up  3:01,  1 user,  load average: 0.87, 0.74, 0.65

58 processes: 56 sleeping, 2 running, 0 zombie, 0 stopped

 

CPU states:       cpu       user      nice      system    irq
softirq  iowait    idle

           total       14.8%   0.0%    5.2%    1.8%    6.5%   0.2%
71.3%

           cpu00   0.0%    0.0%    0.0%     7.0%    26.0%   0.0%
66.9%

           cpu01   42.4%  0.0%    15.4%   0.2%     0.0%    0.2%
41.8%

           cpu02    0.0%    0.0%    0.0%     0.0%     0.0%    0.0%
100.0%

           cpu03   17.0%   0.0%    5.6%     0.0%     0.0%    0.6%
76.8%

 

Mem:  3998688k av, 2519888k used, 1478800k free,       0k shrd,   90812k
buff

           411456k active,            1974520k inactive

Swap: 2040244k av,       0k used, 2040244k free                 2173236k
cached

 

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU
COMMAND

 6982 root      25   0 53620  50M  1192 R    18.6  1.2  45:36   1 snort

 

------------------------------------------------------------------------
------------------------------------------

 

TCPDUMP Statistics after about a 5 min dump at night.

 

13559511 packets received by filter

13503328 packets dropped by kernel

 

 

------------------------------------------------------------------------
------------------------------------------

 

 

INTERFACE statistics after 30 seconds.

 

Interface           Received           Sent     Total

                        Kbps                 Kbps     Kbps

 

eth2                  863163.60        0.00      863163.60

 |---- Max          866570.72        0.00      866570.72

 |---- Avg           831803.76        0.00      831803.76

 

 

 

------------------------------------------------------------------------
------------------------------------------

 

SNORT after 30 seconds:

 

========================================================================
=======

Snort analyzed 2930003 out of 5366187 packets, dropping 2436184(45.399%)
packets

 

Breakdown by protocol:                Action Stats:

    TCP: 491713     (9.163%)          ALERTS: 181       

    UDP: 1633       (0.030%)          LOGGED: 181       

   ICMP: 65         (0.001%)          PASSED: 0         

    ARP: 91         (0.002%)

  EAPOL: 0          (0.000%)

   IPv6: 0          (0.000%)

    IPX: 0          (0.000%)

  OTHER: 274        (0.005%)

DISCARD: 0          (0.000%)

========================================================================
=======

Wireless Stats:

Breakdown by type:

    Management Packets: 0          (0.000%)

    Control Packets:    0          (0.000%)

    Data Packets:       0          (0.000%)

========================================================================
=======

Fragmentation Stats:

Fragmented IP Packets: 3          (0.000%)

    Fragment Trackers: 2         

   Rebuilt IP Packets: 0         

   Frag elements used: 0         

Discarded(incomplete): 0         

   Discarded(timeout): 0         

  Frag2 memory faults: 0         

========================================================================
=======

TCP Stream Reassembly Stats:

        TCP Packets Used: 491713     (9.163%)

         Stream Trackers: 1000      

          Stream flushes: 72        

           Segments used: 257       

   Stream4 Memory Faults: 0         

========================================================================
=======

Final Flow Statistics

,----[ FLOWCACHE STATS ]----------

Memcap: 10485760 Overhead Bytes 16400 used(%1.906099)/blocks
(199869/1284) Overhead blocks: 1 Could Hold: (73326)

IPV4 count: 1283 frees: 0 low_time: 1092189785, high_time: 1092189816,
diff: 0h:00:31s

    finds: 493492 reversed: 176397(%35.744652) 

    find_sucess: 492209 find_fail: 1283 percent_success: (%99.740016)
new_flows: 1283

 Protocol: 1 (%0.013171) finds: 65  reversed: 5(%7.692308) 

  find_sucess: 20 find_fail: 45 percent_success: (%30.769231) new_flows:
45

 Protocol: 6 (%99.654098) finds: 491785  reversed: 176168(%35.822158) 

  find_sucess: 490786 find_fail: 999 percent_success: (%99.796862)
new_flows: 999

 Protocol: 17 (%0.330907) finds: 1633  reversed: 224(%13.717085) 

  find_sucess: 1401 find_fail: 232 percent_success: (%85.793019)
new_flows: 232

 Protocol: 89 (%0.001621) finds: 8  reversed: 0(%0.000000) 

  find_sucess: 2 find_fail: 6 percent_success: (%25.000000) new_flows: 6

 Protocol: 103 (%0.000203) finds: 1  reversed: 0(%0.000000) 

  find_sucess: 0 find_fail: 1 percent_success: (%0.000000) new_flows: 1

database: Closing connection to database ""

Snort exiting