Re: Snort Timestamps Out of Sequence

["ROTNEMER, ALAN H via Snort-devel" <snort-devel () lists snort org>]

Joel,

Our customer was running into the 7-hour delay. We have a daemon process that launches Snort and, via a Fifo pipe, 
sends packets to Snort that were read in by another process that reads packets off the Interface card.
We get the alerts back using:
       output alert_unified2: filename $OUT_FILE, vlan_event_types
The alerts eventually make their way to either a MySQL or Hadoop database. When we attempt the Hadoop ingestion, we 
spot an alert whose timestamp is 7 hours prior to the previous one, and we log an snmp trap.

When this problem was reported,  I saw the way to create the ALERT_CSV file, and I saw that while most of the output 
came in timestamp order, there were numerous instances where there were the “late” ones.

I took our product out of the picture by running a separate instance of Snort that got its input directly from the 
Interface card. The delays weren’t quite as long – up to 30 minutes, but they happened.  I ran this instance for about 
2 hours. (The customer reported the 6-hour delay after running Snort for a couple of days).

Let me know if you need anything else to help diagnose this.

Alan





From: Joel Esler (jesler) <jesler () cisco com>
Sent: Wednesday, May 22, 2019 4:11 PM
To: ROTNEMER, ALAN H <ar435f () att com>
Cc: snort-devel () lists snort org
Subject: Re: [Snort-devel] Snort Timestamps Out of Sequence

Alan,

7 hours certainly seems incorrect.  What is the output method?  Syslog?  Are you doing syslog output directly from 
Snort, or from barnyard2?



From: "ROTNEMER, ALAN H" <ar435f () att com<mailto:ar435f () att com>>
Date: Wednesday, May 22, 2019 at 3:14 PM
To: "Joel Esler (jesler)" <jesler () cisco com<mailto:jesler () cisco com>>
Cc: "snort-devel () lists snort org<mailto:snort-devel () lists snort org>" <snort-devel () lists snort 
org<mailto:snort-devel () lists snort org>>
Subject: RE: [Snort-devel] Snort Timestamps Out of Sequence

Hey Joel,

After looking at our packets and discussing this with my development and analyst groups, I guess there is just one 
thing I need to know:

From what you have said in your replies, there appears to be “conditions” where Snort will receive a packet, and, for 
whatever reasons, delay sending the alert. It could be something about the packet, or something about the rule. My 
customer has had instances of delays of up to 7 hours.  This ends up causing a bit of a problem on our backend.

Our product that processes the alerts coming from Snort does not expect these delays. In fact, until this issue 
appeared, we always thought that the alerts would arrive in the (timestamp) order they were fed into Snort.

Can you confirm, then, that Snort COULD delay alerts for some packets, and, thus, it is possible that alerts will not 
be returned in timestamp order?

If yes, are there known circumstances where this could occur, and would you be able to document them for us? Or 
describe the processing that occurs within Snort that could lead to this situation?

We can make some adjustments to our backend, but I want to be able to explain this to my development group and our 
customer.

Many thanks,
Alan
P.S. If you know of someone else I need to contact, could you let me know?





From: Joel Esler (jesler) <jesler () cisco com<mailto:jesler () cisco com>>
Sent: Monday, May 6, 2019 9:44 AM
To: ROTNEMER, ALAN H <ar435f () att com<mailto:ar435f () att com>>
Cc: snort-devel () lists snort org<mailto:snort-devel () lists snort org>
Subject: Re: [Snort-devel] Snort Timestamps Out of Sequence

Hey Alan,

My "off the cuff" theory, without looking at your Snort configuration and requesting a full traffic reassembly is that 
something was holding the connection open (for 7 minutes) (keep-alive?) and Snort is reassembling the HTTP session in 
the background into what we call a "pseudo" packet.  A large reassembled stream.  That's what your rule alerted on, and 
should have logged it to disk.




--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.talosintelligence.com&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=SF14kolGOfb9ES7hY9eg3w&m=cN5KAl1AsQe0jtiNGpL0uP5-1hkgqFQNgRED18eFkm4&s=29AEhc-8fQJZvQEt2b0d4cmzF8_G5pWBHKOucV-PHaQ&e=>


On May 6, 2019, at 9:16 AM, ROTNEMER, ALAN H <ar435f () att com<mailto:ar435f () att com>> wrote:

Is there some explanation as to why the alert took over 7 minutes to publish? Could Snort be waiting on anything in 
order to complete the alert?

_______________________________________________
Snort-devel mailing list
Snort-devel () lists snort org
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org for the latest news about Snort!