Snort mailing list archives
Re: File preprocessor fails to capture files
From: Pablo Cantos Polaino <pcantos () redborder org>
Date: Fri, 8 May 2015 21:29:48 +0200
Exit stats when listening from interface:
===============================================================================
Run time for packet processing was 86.342415 seconds
Snort processed 247599 packets.
Snort ran for 0 days 0 hours 1 minutes 26 seconds
Pkts/min: 247599
Pkts/sec: 2879
===============================================================================
Memory usage summary:
Total non-mmapped bytes (arena): 10100736
Bytes in mapped regions (hblkhd): 122081280
Total allocated space (uordblks): 8073952
Total free space (fordblks): 2026784
Topmost releasable block (keepcost): 108528
===============================================================================
Packet I/O Totals:
Received: 247599
Analyzed: 247599 (100.000%)
Dropped: 0 ( 0.000%)
Filtered: 0 ( 0.000%)
Outstanding: 0 ( 0.000%)
Injected: 0
===============================================================================
Breakdown by protocol (includes rebuilt packets):
Eth: 247605 (100.000%)
VLAN: 0 ( 0.000%)
IP4: 247503 ( 99.959%)
Frag: 0 ( 0.000%)
ICMP: 0 ( 0.000%)
UDP: 24 ( 0.010%)
TCP: 125325 ( 50.615%)
IP6: 0 ( 0.000%)
IP6 Ext: 0 ( 0.000%)
IP6 Opts: 0 ( 0.000%)
Frag6: 0 ( 0.000%)
ICMP6: 0 ( 0.000%)
UDP6: 0 ( 0.000%)
TCP6: 0 ( 0.000%)
Teredo: 0 ( 0.000%)
ICMP-IP: 0 ( 0.000%)
IP4/IP4: 0 ( 0.000%)
IP4/IP6: 0 ( 0.000%)
IP6/IP4: 0 ( 0.000%)
IP6/IP6: 0 ( 0.000%)
GRE: 0 ( 0.000%)
GRE Eth: 0 ( 0.000%)
GRE VLAN: 0 ( 0.000%)
GRE IP4: 0 ( 0.000%)
GRE IP6: 0 ( 0.000%)
GRE IP6 Ext: 0 ( 0.000%)
GRE PPTP: 0 ( 0.000%)
GRE ARP: 0 ( 0.000%)
GRE IPX: 0 ( 0.000%)
GRE Loop: 0 ( 0.000%)
MPLS: 0 ( 0.000%)
ARP: 102 ( 0.041%)
IPX: 0 ( 0.000%)
Eth Loop: 0 ( 0.000%)
Eth Disc: 0 ( 0.000%)
IP4 Disc: 122145 ( 49.331%)
IP6 Disc: 0 ( 0.000%)
TCP Disc: 0 ( 0.000%)
UDP Disc: 0 ( 0.000%)
ICMP Disc: 0 ( 0.000%)
All Discard: 122145 ( 49.331%)
Other: 9 ( 0.004%)
Bad Chk Sum: 379 ( 0.153%)
Bad TTL: 0 ( 0.000%)
S5 G 1: 2 ( 0.001%)
S5 G 2: 4 ( 0.002%)
Total: 247605
===============================================================================
Action Stats:
Alerts: 0 ( 0.000%)
Logged: 0 ( 0.000%)
Passed: 0 ( 0.000%)
Limits:
Match: 0
Queue: 0
Log: 0
Event: 0
Alert: 0
Verdicts:
Allow: 215292 ( 86.952%)
Block: 0 ( 0.000%)
Replace: 0 ( 0.000%)
Whitelist: 32307 ( 13.048%)
Blacklist: 0 ( 0.000%)
Ignore: 0 ( 0.000%)
===============================================================================
Frag3 statistics:
Total Fragments: 0
Frags Reassembled: 0
Discards: 0
Memory Faults: 0
Timeouts: 0
Overlaps: 0
Anomalies: 0
Alerts: 0
Drops: 0
FragTrackers Added: 0
FragTrackers Dumped: 0
FragTrackers Auto Freed: 0
Frag Nodes Inserted: 0
Frag Nodes Deleted: 0
===============================================================================
===============================================================================
Stream statistics:
Total sessions: 20
TCP sessions: 14
UDP sessions: 6
ICMP sessions: 0
IP sessions: 0
TCP Prunes: 0
UDP Prunes: 0
ICMP Prunes: 0
IP Prunes: 0
TCP StreamTrackers Created: 14
TCP StreamTrackers Deleted: 14
TCP Timeouts: 0
TCP Overlaps: 0
TCP Segments Queued: 6942
TCP Segments Released: 6942
TCP Rebuilt Packets: 6267
TCP Segments Used: 6919
TCP Discards: 48
TCP Gaps: 6459
UDP Sessions Created: 6
UDP Sessions Deleted: 6
UDP Timeouts: 0
UDP Discards: 0
Events: 17
Internal Events: 0
TCP Port Filter
Filtered: 0
Inspected: 0
Tracked: 124952
UDP Port Filter
Filtered: 0
Inspected: 0
Tracked: 6
===============================================================================
HTTP Inspect - encodings (Note: stream-reassembled packets included):
POST methods: 0
GET methods: 0
HTTP Request Headers extracted: 0
HTTP Request Cookies extracted: 0
Post parameters extracted: 0
HTTP response Headers extracted: 2
HTTP Response Cookies extracted: 0
Unicode: 0
Double unicode: 0
Non-ASCII representable: 0
Directory traversals: 0
Extra slashes ("//"): 0
Self-referencing paths ("./"): 0
HTTP Response Gzip packets extracted: 0
Gzip Compressed Data Processed: n/a
Gzip Decompressed Data Processed: n/a
Total packets processed: 13159
===============================================================================
SMTP Preprocessor Statistics
Total sessions : 0
Max concurrent sessions : 0
===============================================================================
dcerpc2 Preprocessor Statistics
Total sessions: 0
===============================================================================
SSL Preprocessor:
SSL packets decoded: 14
Client Hello: 0
Server Hello: 2
Certificate: 2
Server Done: 3
Client Key Exchange: 0
Server Key Exchange: 0
Change Cipher: 3
Finished: 0
Client Application: 0
Server Application: 1
Alert: 0
Unrecognized records: 9
Completed handshakes: 0
Bad handshakes: 0
Sessions ignored: 1
Detection disabled: 2
===============================================================================
SIP Preprocessor Statistics
Total sessions: 0
===============================================================================
File Preprocessor Statistics
Total file type callbacks: 0
Total file signature callbacks: 1
Total files would saved to disk: 1
Total files saved to disk: 1
Total file data saved to disk: 446 bytes
Total files duplicated: 0
Total files reserving failed: 0
Total file capture min: 0
Total file capture max: 0
Total file capture memcap: 0
Total files reading failed: 0
Total file agent memcap failures: 0
Total files sent: 0
Total file data sent: 0
Total file transfer failures: 0
===============================================================================
File type stats:
Type Download (Bytes) Upload (Bytes)
Total 0 0 0 0
File signature stats:
Type Download Upload
Undecided file type, continue...( 0) 1 0
Total 1 0
File type verdicts:
UNKNOWN: 0
LOG: 0
STOP: 0
BLOCK: 0
REJECT: 0
PENDING: 0
STOP CAPTURE: 0
Total: 0
File signature verdicts:
UNKNOWN: 1
LOG: 0
STOP: 0
BLOCK: 0
REJECT: 0
PENDING: 0
STOP CAPTURE: 0
Total: 1
Total files processed: 2
Total files data processed: 2594891 bytes
Total files buffered: 2
Total files released: 1
Total files freed: 1
Total files captured: 1
Total files within one packet: 1
Total buffers allocated: 81
Total buffers freed: 80
Total buffers released: 1
Maximum file buffers used: 80
Total buffers free errors: 0
Total buffers release errors: 0
Total memcap failures: 0
Total memcap failures at reserve: 0
Total reserve failures: 0
Total file capture size min: 0
Total file capture size max: 0
Total capture max before reserve: 0
Total file signature max: 0
Maximum buffers can allocate: 3196
Number of buffers in use: 0
Number of buffers in free list: 3195
Number of buffers in release list: 1
===============================================================================
Snort exiting
###################################################################################
###################################################################################
Exit stats when reading the PCAP file:
===============================================================================
Run time for packet processing was 3.962580 seconds
Snort processed 3326 packets.
Snort ran for 0 days 0 hours 0 minutes 3 seconds
Pkts/sec: 1108
===============================================================================
Memory usage summary:
Total non-mmapped bytes (arena): 10190848
Bytes in mapped regions (hblkhd): 122081280
Total allocated space (uordblks): 8072912
Total free space (fordblks): 2117936
Topmost releasable block (keepcost): 132992
===============================================================================
Packet I/O Totals:
Received: 3326
Analyzed: 3326 (100.000%)
Dropped: 0 ( 0.000%)
Filtered: 0 ( 0.000%)
Outstanding: 0 ( 0.000%)
Injected: 0
===============================================================================
Breakdown by protocol (includes rebuilt packets):
Eth: 3333 (100.000%)
VLAN: 0 ( 0.000%)
IP4: 3333 (100.000%)
Frag: 0 ( 0.000%)
ICMP: 0 ( 0.000%)
UDP: 40 ( 1.200%)
TCP: 3293 ( 98.800%)
IP6: 0 ( 0.000%)
IP6 Ext: 0 ( 0.000%)
IP6 Opts: 0 ( 0.000%)
Frag6: 0 ( 0.000%)
ICMP6: 0 ( 0.000%)
UDP6: 0 ( 0.000%)
TCP6: 0 ( 0.000%)
Teredo: 0 ( 0.000%)
ICMP-IP: 0 ( 0.000%)
IP4/IP4: 0 ( 0.000%)
IP4/IP6: 0 ( 0.000%)
IP6/IP4: 0 ( 0.000%)
IP6/IP6: 0 ( 0.000%)
GRE: 0 ( 0.000%)
GRE Eth: 0 ( 0.000%)
GRE VLAN: 0 ( 0.000%)
GRE IP4: 0 ( 0.000%)
GRE IP6: 0 ( 0.000%)
GRE IP6 Ext: 0 ( 0.000%)
GRE PPTP: 0 ( 0.000%)
GRE ARP: 0 ( 0.000%)
GRE IPX: 0 ( 0.000%)
GRE Loop: 0 ( 0.000%)
MPLS: 0 ( 0.000%)
ARP: 0 ( 0.000%)
IPX: 0 ( 0.000%)
Eth Loop: 0 ( 0.000%)
Eth Disc: 0 ( 0.000%)
IP4 Disc: 0 ( 0.000%)
IP6 Disc: 0 ( 0.000%)
TCP Disc: 0 ( 0.000%)
UDP Disc: 0 ( 0.000%)
ICMP Disc: 0 ( 0.000%)
All Discard: 0 ( 0.000%)
Other: 0 ( 0.000%)
Bad Chk Sum: 0 ( 0.000%)
Bad TTL: 0 ( 0.000%)
S5 G 1: 3 ( 0.090%)
S5 G 2: 4 ( 0.120%)
Total: 3333
===============================================================================
Action Stats:
Alerts: 0 ( 0.000%)
Logged: 0 ( 0.000%)
Passed: 0 ( 0.000%)
Limits:
Match: 0
Queue: 0
Log: 0
Event: 0
Alert: 0
Verdicts:
Allow: 3326 (100.000%)
Block: 0 ( 0.000%)
Replace: 0 ( 0.000%)
Whitelist: 0 ( 0.000%)
Blacklist: 0 ( 0.000%)
Ignore: 0 ( 0.000%)
===============================================================================
Frag3 statistics:
Total Fragments: 0
Frags Reassembled: 0
Discards: 0
Memory Faults: 0
Timeouts: 0
Overlaps: 0
Anomalies: 0
Alerts: 0
Drops: 0
FragTrackers Added: 0
FragTrackers Dumped: 0
FragTrackers Auto Freed: 0
Frag Nodes Inserted: 0
Frag Nodes Deleted: 0
===============================================================================
===============================================================================
Stream statistics:
Total sessions: 24
TCP sessions: 14
UDP sessions: 10
ICMP sessions: 0
IP sessions: 0
TCP Prunes: 0
UDP Prunes: 0
ICMP Prunes: 0
IP Prunes: 0
TCP StreamTrackers Created: 14
TCP StreamTrackers Deleted: 14
TCP Timeouts: 0
TCP Overlaps: 0
TCP Segments Queued: 1895
TCP Segments Released: 1895
TCP Rebuilt Packets: 1304
TCP Segments Used: 1894
TCP Discards: 0
TCP Gaps: 0
UDP Sessions Created: 10
UDP Sessions Deleted: 10
UDP Timeouts: 0
UDP Discards: 0
Events: 1
Internal Events: 0
TCP Port Filter
Filtered: 0
Inspected: 0
Tracked: 3286
UDP Port Filter
Filtered: 0
Inspected: 0
Tracked: 10
===============================================================================
HTTP Inspect - encodings (Note: stream-reassembled packets included):
POST methods: 0
GET methods: 10
HTTP Request Headers extracted: 10
HTTP Request Cookies extracted: 0
Post parameters extracted: 0
HTTP response Headers extracted: 10
HTTP Response Cookies extracted: 0
Unicode: 0
Double unicode: 0
Non-ASCII representable: 0
Directory traversals: 0
Extra slashes ("//"): 0
Self-referencing paths ("./"): 0
HTTP Response Gzip packets extracted: 0
Gzip Compressed Data Processed: n/a
Gzip Decompressed Data Processed: n/a
Total packets processed: 2944
===============================================================================
SMTP Preprocessor Statistics
Total sessions : 0
Max concurrent sessions : 0
===============================================================================
dcerpc2 Preprocessor Statistics
Total sessions: 0
===============================================================================
===============================================================================
SIP Preprocessor Statistics
Total sessions: 0
===============================================================================
File Preprocessor Statistics
Total file type callbacks: 0
Total file signature callbacks: 10
Total files would saved to disk: 10
Total files saved to disk: 10
Total file data saved to disk: 47473897 bytes
Total files duplicated: 0
Total files reserving failed: 0
Total file capture min: 0
Total file capture max: 0
Total file capture memcap: 0
Total files reading failed: 0
Total file agent memcap failures: 0
Total files sent: 0
Total file data sent: 0
Total file transfer failures: 0
===============================================================================
File type stats:
Type Download (Bytes) Upload (Bytes)
Total 0 0 0 0
File signature stats:
Type Download Upload
Undecided file type, continue...( 0) 10 0
Total 10 0
File type verdicts:
UNKNOWN: 0
LOG: 0
STOP: 0
BLOCK: 0
REJECT: 0
PENDING: 0
STOP CAPTURE: 0
Total: 0
File signature verdicts:
UNKNOWN: 10
LOG: 0
STOP: 0
BLOCK: 0
REJECT: 0
PENDING: 0
STOP CAPTURE: 0
Total: 10
Total files processed: 10
Total files data processed: 47473024 bytes
Total files buffered: 10
Total files released: 10
Total files freed: 0
Total files captured: 10
Total files within one packet: 4
Total buffers allocated: 1455
Total buffers freed: 0
Total buffers released: 1455
Maximum file buffers used: 787
Total buffers free errors: 0
Total buffers release errors: 0
Total memcap failures: 0
Total memcap failures at reserve: 0
Total reserve failures: 0
Total file capture size min: 0
Total file capture size max: 0
Total capture max before reserve: 0
Total file signature max: 0
Maximum buffers can allocate: 3196
Number of buffers in use: 0
Number of buffers in free list: 1741
Number of buffers in release list: 1455
===============================================================================
Snort exiting
Pablo Cantos
redborder.org / pcantos () redborder org
2015-05-08 15:26 GMT+02:00 Hui cao <huica () cisco com>:
What's the exit stats? Best, Hui. On 05/08/2015 08:58 AM, Pablo Cantos Polaino wrote: Thanks for your reply Hui, I'm attaching the full configuration now. I've used a default conf, and included the file preprocessor configuration that I mentioned before. As you can see in the conf file, for normalize preprocessor, there was the following line in the default conf, so I suppose I shouldn't change this: preprocessor normalize_tcp: ips ecn stream About debug, I haven't build snort in debug mode since I haven't be able to go deeper into this. I will try this when I come back to the office, but in any case, I'm interested on use Snort in a normal mode, not in debug mode. I forgot to mention I'm using the last version: 2.9.7.2. Best Regards, Pablo Cantos redborder.org / pcantos () redborder org 2015-05-08 14:40 GMT+02:00 Hui Cao (huica) <huica () cisco com>:What’s the full snort configuration? If you build snort with debug, you should add:config paf_max: 16384 In addition, it would be better to add: preprocessor normalize_tcp: ips <https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=0CCQQFjAC&url=http%3A%2F%2Ft73100.security-ids-snort-general.securityupdate.info%2Fpreprocessor-normalize-tcp-ips-t73100.html&ei=B65MVdGDEJObyAT5g4GQBg&usg=AFQjCNEvwb_tSISxggsZbXdfA2SJs7Pm1A&sig2=0_WSEYBph2TfDNTtcatjhw> Best, Hui. From: Pablo Cantos Polaino <pcantos () redborder org> Date: Friday, May 8, 2015 at 8:26 AM To: "snort-users () lists sourceforge net" < snort-users () lists sourceforge net> Subject: [Snort-users] File preprocessor fails to capture files Hello all, I'm doing some tests over the file preprocessor and these are the conf options that I'm using related to file preprocessor: include file_magic.confconfig file:\ file_type_depth 4294967295, \ file_signature_depth 4294967295, \ file_capture_max 4294967295 preprocessor file_inspect:\ capture_queue_size 50000, \ signature, \ capture_disk /var/log/snort/files/ 50000This time what I'm trying to do is to capture every file detected by file preprocessor in the directory /var/log/snort/files. For these tests, I've used the following files: wget ftp://ftp.hp.com/pub/information_storage/software/video/video1.avi wget ftp://ftp.hp.com/pub/information_storage/software/video/MakeUp.mov wget ftp://ftp.hp.com/pub/information_storage/software/video/Fighter.mpg wget http://releases.ubuntu.com/14.04/ubuntu-14.04.2-desktop-amd64.iso wget http://scholar.princeton.edu/sites/default/files/oversize_pdf_test_0.pdf wget https://10.0.70.110/client/VMware-viclient.exe --no-check-certificate wget http://cpansearch.perl.org/src/MIKEM/Device-SNP-1.3/datadesigner/tux-sw.bmp I addition, I've got a pcap traffic capture which includes all the 7 files above. When I run Snort reading this pcap, I got the following: Captured files:# ls -lS-rw------- 1 root root 24211979 May 8 11:148452B621DC334D1FD44470A80540CBEF2F6869AF851B9E8C684EF9402016F692 -rw------- 1 root root 13045613 May 8 11:14 5CF142947C2957EE648457A91B69FB82F088F31205030F9A77B2AD827228C6E9 -rw------- 1 root root 6352738 May 8 11:14 DB57C532919D9ABABAC127F29DBDC05ED832394880E46CAD81A5DDE713CCB4BE -rw------- 1 root root 2936119 May 8 11:14 B4127F43A3F455523B81179CC11AA4F28FC27F4C041D20E28AA08A32D85CB757 -rw------- 1 root root 495316 May 8 11:14 A294AA3D01CD8902BF842D320E7F2C043AF9EAD95D0E7198C3B71A0DBC9D253C -rw------- 1 root root 424526 May 8 11:14 8863DB1EC4B02D5BCC1FB4BD03D220F7458136342CDD47CE507A5B886C6BB56C -rw------- 1 root root 2817 May 8 11:14 D03CDB1F2584A2C06E866931EC5F31F141D9D08F237E04708C7C19D94FFA62F5 -rw------- 1 root root 1958 May 8 11:14 369FDD6FB34BB5E1F0EC79D063FE0115AEF35AA20972BE8E4739417594F692AA -rw------- 1 root root 1958 May 8 11:14 EF49069F43D349C83873A6784351F16ADC39B8358ACFAE3A30EA4DD684C29DCC-rw------- 1 root root 446 May 8 11:148D490C71A27631CF6A476F68C409655CB63BF32C17846A3C3C125A79046DB2C1Downloaded files:# ls -l-rw-r--r-- 1 root root 2187725 May 8 11:01 Fighter.mpg-rw-r--r-- 1 root root 14955972 May 8 11:01 MakeUp.mov-rw-r--r-- 1 root root 375187792 May 8 11:02 VMware-viclient.exe -rw-r--r-- 1 root root 101688487 Jul 10 2014 oversize_pdf_test_0.pdf-rw-r--r-- 1 root root 446 Mar 22 2013 tux-sw.bmp -rw-r--r-- 1 root root 1044381696 Feb 18 20:12ubuntu-14.04.2-desktop-amd64.iso -rw-r--r-- 1 root root 6094376 May 8 11:01 video1.avi # sha256sum *55bdca20aa0ffd8fa3b12029d1e122696a936abc29dd4ec4a5bd878836a5d36fFighter.mpg88a43830b006a4ade60874ffb10a0d5afd06245d0bc460da90015ed73df08d58MakeUp.mov 57bc6123a563056e32fb317c20d1e3b96af723b2b2c9732033e3ab9ce8f8e625 VMware-viclient.exe fa43e683e94372d81210a275cc37112bf2df9c971d377506aab8ae47e5fb0d34 oversize_pdf_test_0.pdf 8d490c71a27631cf6a476f68c409655cb63bf32c17846a3c3c125a79046db2c1 tux-sw.bmp39eeb28bdb8af630850e75e54b9864ca07640a2bb10bd10055763236b99f9b1dubuntu-14.04.2-desktop-amd64.iso bb13418aeb4535c0d1f5c491ad69dd87041a8a1ba7dacc6bc763337beaed7dca video1.aviAs you can see, Snort just captures correctly the smallest file, that fits in a single packet. The others captured files do not coincide with the captured files (in number and size, and hence in sha256) If I run Snort sniffing from my network interface and I download the 7 files by using the wget command, I got the following: Captured files:-rw------- 1 root root 446 May 8 11:308D490C71A27631CF6A476F68C409655CB63BF32C17846A3C3C125A79046DB2C1This case, Snort just captures the smallest file, that fits in a single packet. I've gone deep into the code and I've found out the problem could come from a strange behavior of the Frag3 preprocessor when dealing with packets that contain files. I see two different issues here: 1.- When sniffing from an interface, Snort is only able to capture files which fit in one single packet. 2.- When reading from a network capture file, Snort is able to capture files in general, but it does it in a wrong way when the file take up more than one packet. I'd like to know if you were aware of these strange behaviors. Best Regards, Pablo Cantos redborder.org / pcantos () redborder org
------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users Please visit http://blog.snort.org to stay current on all the latest Snort news!
Current thread:
- File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui Cao (huica) (May 08)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui cao (May 08)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui Cao (huica) (May 08)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui Cao (huica) (May 08)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 15)
- Re: File preprocessor fails to capture files Russ (May 15)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 16)
- Re: File preprocessor fails to capture files Russ (May 17)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 18)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui Cao (huica) (May 08)