Re: Snort spikes to 100% CPU followed by network latency

[Cody Brugh <cbrugh () gmail com>]

Ok, who can examine the core files once I have them?



On Wed, May 28, 2014 at 5:43 PM, Russ Combs (rucombs) <rucombs () cisco com>wrote:

>  ------------------------------
> *From:* Cody Brugh [cbrugh () gmail com]
> *Sent:* Wednesday, May 28, 2014 5:40 PM
>
> *To:* Russ Combs (rucombs)
> *Cc:* snort-users () lists sourceforge net
> *Subject:* Re: [Snort-users] Snort spikes to 100% CPU followed by network
> latency
>
>   Also note that when we see these CPU/latency spikes we have no alerts
> or drops that would easily tell us what is causing the problem. If it's not
> a rule what should I start turning off to try eliminate possible causes?
>  It's something that doesn't log or anything.
>
>  * Another option you have is to compile with debug and generate a core
> by sending a sig abort to the Snort process when it is in the 100% CPU
> state.  And you should capture a few to ensure they are in the same basic
> area.  Then the cores need to be examined for clues.
>
>
> On May 28, 2014, at 5:12 PM, "Russ Combs (rucombs)" <rucombs () cisco com>
> wrote:
>
>
>  ------------------------------
> *From:* Cody Brugh [cbrugh () gmail com]
> *Sent:* Tuesday, May 27, 2014 6:30 PM
> *To:* Russ Combs (rucombs)
> *Cc:* snort-users () lists sourceforge net
> *Subject:* Re: [Snort-users] Snort spikes to 100% CPU followed by network
> latency
>
>   Russ,
>
>  I am still having latency/CPU spike issues even after enabling PPM
> configuration.... In the below logs where does it tell me which rule SIG ID
> so I can disable the rules that are causing me slowness?
>
>  * Actually, it doesn't tell you which rule.  Only which rule tree, which
> isn't terribly helpful.  That's because the rules are compiled into a
> different form to eliminate redundant checks.
>
>  It does tell you that the rule trees that are triggering the PPM events
> are around 55 usec although you have one shown which is higher.  What is
> your threshold set to?
>
>  Also, did you analyze the packets logged with the events to see what
> type of traffic this is?  You may be able to narrow it down and capture a
> whole session that triggers the problem and go from there.
>
>
> PPM: Rule-Event Pkt[20570] address=0x0x8200f80 re-enabled
> 05/27-18:26:27.989936
> PPM: Rule-Event Pkt[20570] suspended (72.194.88.31:53484 -> 10.2.14.21:80
> ).
> PPM: Rule-Event Pkt[20570] address=0x0x8200f80 used=52.7766 usecs
> suspended 05/27-18:26:27.989936
> PPM: Rule-Event Pkt[30776] address=0x0x3c28fb0 re-enabled
> 05/27-18:26:36.195293
> PPM: Rule-Event Pkt[36115] suspended (10.2.13.17:80 ->
> 164.113.217.51:41204).
> PPM: Rule-Event Pkt[36115] address=0x0x3c28fb0 used=54.8616 usecs
> suspended 05/27-18:26:40.813898
> PPM: Rule-Event Pkt[47155] address=0x0x8200f80 re-enabled
> 05/27-18:26:48.038002
> PPM: Rule-Event Pkt[48185] suspended (66.87.133.205:4163 -> 10.2.2.4:80).
> PPM: Rule-Event Pkt[48185] address=0x0x8200f80 used=53.992 usecs suspended
> 05/27-18:26:49.081371
> PPM: Rule-Event Pkt[60578] address=0x0x3c28fb0 re-enabled
> 05/27-18:27:00.867186
> PPM: Rule-Event Pkt[71509] suspended (10.2.13.48:80 -> 96.44.123.180:1046
> ).
> PPM: Rule-Event Pkt[71509] address=0x0x3c28fb0 used=54.4075 usecs
> suspended 05/27-18:27:08.522285
> PPM: Rule-Event Pkt[72989] address=0x0x8200f80 re-enabled
> 05/27-18:27:09.766234
> PPM: Rule-Event Pkt[76364] suspended (10.2.13.1:35718 -> 66.135.58.62:80).
> PPM: Rule-Event Pkt[76364] address=0x0x8200f80 used=76.2302 usecs
> suspended 05/27-18:27:15.130825
> PPM: Rule-Event Pkt[77634] suspended (66.8.180.174:64983 -> 10.2.13.17:80
> ).
> PPM: Rule-Event Pkt[77634] address=0x0x820c0b0 used=53.126 usecs suspended
> 05/27-18:27:17.180899
>
>
>
>
> On Fri, May 23, 2014 at 8:09 AM, Russ Combs (rucombs) <rucombs () cisco com>wrote:
>
> >  ------------------------------
> > *From:* Cody Brugh [cbrugh () gmail com]
> > *Sent:* Thursday, May 22, 2014 8:13 PM
> > *To:* snort-users () lists sourceforge net
> > *Subject:* [Snort-users] Snort spikes to 100% CPU followed by network
> > latency
> >
> >       Hello,
> >
> >  We have been running snort in-line for over a year now with no issues in
> > terms of latency or CPU usage.  Recently (over the past month) snort will
> > all of the sudden spike CPU usage up to 100% and network latency becomes
> > real bad, 1000+ms.
> >
> >  I am really not sure where to start on figuring out what is causing
> > this.  I am starting snort so it prints the alerts/drops on the console and
> > don't see any specific rule that would be causing this.
> >
> >  Any advise on this issue?
> >
> >  * Did you change your Snort version or configuration around the time
> > you started seeing the issue?  How frequently does this occur?  And when it
> > happens does it resolve itself or do you restart or what?
> >
> >  You can turn on PPM (config ppm ...) and enable the PPM rules (gid
> > 134).  That may catch the problem packet which you can log and examine for
> > clues.
> >
> >  Without any clues I'd first check for SDF and PCRE.  If you have SDF
> > (preprocessor sensitive_data) configured you can try commenting that out.
> >  If you have any pcre/O rules (PCRE override) you can try commenting those
> > out too.
> >
> >  Snort OS: CentOS, 64-bit
> >
> >   o"  )~   Version 2.9.6.1 GRE (Build 56)
> >    ''''    By Martin Roesch & The Snort Team:
> > http://www.snort.org/snort/snort-team
> >            Copyright (C) 2014 Cisco and/or its affiliates. All rights
> > reserved.
> >            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
> >            Using libpcap version 1.0.0
> >            Using PCRE version: 7.8 2008-09-05
> >            Using ZLIB version: 1.2.3
> >
> > DAQ version: 2.0.2
> >
> >  Thanks!
------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download
http://p.sf.net/sfu/restlet

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!