Snort mailing list archives
Re: File preprocessor fails to capture files
From: Pablo Cantos Polaino <pcantos () redborder org>
Date: Fri, 8 May 2015 23:59:42 +0200
Hi Hui,
I've replaced config paf_max: 16000 by 60000 as you propose.
File type was not identified because I had disabled type_id option in
preprocessor file_inspect. I've replayed the tests with paf_max = 60000 and
both type_id enabled and disabled. In both cases the capture files are the
same (number and size) when sniffering from an interface and reading from a
PCAP file. So I'm pasting below the exit stats when type_id is enabled and
paf_max = 60000:
Exit stats when reading the PCAP file and type_id enabled:
===============================================================================
Run time for packet processing was 3.978146 seconds
Snort processed 3326 packets.
Snort ran for 0 days 0 hours 0 minutes 3 seconds
Pkts/sec: 1108
===============================================================================
Memory usage summary:
Total non-mmapped bytes (arena): 10190848
Bytes in mapped regions (hblkhd): 122081280
Total allocated space (uordblks): 8072896
Total free space (fordblks): 2117952
Topmost releasable block (keepcost): 133008
===============================================================================
Packet I/O Totals:
Received: 3326
Analyzed: 3326 (100.000%)
Dropped: 0 ( 0.000%)
Filtered: 0 ( 0.000%)
Outstanding: 0 ( 0.000%)
Injected: 0
===============================================================================
Breakdown by protocol (includes rebuilt packets):
Eth: 3333 (100.000%)
VLAN: 0 ( 0.000%)
IP4: 3333 (100.000%)
Frag: 0 ( 0.000%)
ICMP: 0 ( 0.000%)
UDP: 40 ( 1.200%)
TCP: 3293 ( 98.800%)
IP6: 0 ( 0.000%)
IP6 Ext: 0 ( 0.000%)
IP6 Opts: 0 ( 0.000%)
Frag6: 0 ( 0.000%)
ICMP6: 0 ( 0.000%)
UDP6: 0 ( 0.000%)
TCP6: 0 ( 0.000%)
Teredo: 0 ( 0.000%)
ICMP-IP: 0 ( 0.000%)
IP4/IP4: 0 ( 0.000%)
IP4/IP6: 0 ( 0.000%)
IP6/IP4: 0 ( 0.000%)
IP6/IP6: 0 ( 0.000%)
GRE: 0 ( 0.000%)
GRE Eth: 0 ( 0.000%)
GRE VLAN: 0 ( 0.000%)
GRE IP4: 0 ( 0.000%)
GRE IP6: 0 ( 0.000%)
GRE IP6 Ext: 0 ( 0.000%)
GRE PPTP: 0 ( 0.000%)
GRE ARP: 0 ( 0.000%)
GRE IPX: 0 ( 0.000%)
GRE Loop: 0 ( 0.000%)
MPLS: 0 ( 0.000%)
ARP: 0 ( 0.000%)
IPX: 0 ( 0.000%)
Eth Loop: 0 ( 0.000%)
Eth Disc: 0 ( 0.000%)
IP4 Disc: 0 ( 0.000%)
IP6 Disc: 0 ( 0.000%)
TCP Disc: 0 ( 0.000%)
UDP Disc: 0 ( 0.000%)
ICMP Disc: 0 ( 0.000%)
All Discard: 0 ( 0.000%)
Other: 0 ( 0.000%)
Bad Chk Sum: 0 ( 0.000%)
Bad TTL: 0 ( 0.000%)
S5 G 1: 3 ( 0.090%)
S5 G 2: 4 ( 0.120%)
Total: 3333
===============================================================================
Action Stats:
Alerts: 0 ( 0.000%)
Logged: 0 ( 0.000%)
Passed: 0 ( 0.000%)
Limits:
Match: 0
Queue: 0
Log: 0
Event: 0
Alert: 0
Verdicts:
Allow: 3326 (100.000%)
Block: 0 ( 0.000%)
Replace: 0 ( 0.000%)
Whitelist: 0 ( 0.000%)
Blacklist: 0 ( 0.000%)
Ignore: 0 ( 0.000%)
===============================================================================
Frag3 statistics:
Total Fragments: 0
Frags Reassembled: 0
Discards: 0
Memory Faults: 0
Timeouts: 0
Overlaps: 0
Anomalies: 0
Alerts: 0
Drops: 0
FragTrackers Added: 0
FragTrackers Dumped: 0
FragTrackers Auto Freed: 0
Frag Nodes Inserted: 0
Frag Nodes Deleted: 0
===============================================================================
===============================================================================
Stream statistics:
Total sessions: 24
TCP sessions: 14
UDP sessions: 10
ICMP sessions: 0
IP sessions: 0
TCP Prunes: 0
UDP Prunes: 0
ICMP Prunes: 0
IP Prunes: 0
TCP StreamTrackers Created: 14
TCP StreamTrackers Deleted: 14
TCP Timeouts: 0
TCP Overlaps: 0
TCP Segments Queued: 2394
TCP Segments Released: 2394
TCP Rebuilt Packets: 793
TCP Segments Used: 2393
TCP Discards: 0
TCP Gaps: 0
UDP Sessions Created: 10
UDP Sessions Deleted: 10
UDP Timeouts: 0
UDP Discards: 0
Events: 1
Internal Events: 0
TCP Port Filter
Filtered: 0
Inspected: 0
Tracked: 3286
UDP Port Filter
Filtered: 0
Inspected: 0
Tracked: 10
===============================================================================
HTTP Inspect - encodings (Note: stream-reassembled packets included):
POST methods: 0
GET methods: 10
HTTP Request Headers extracted: 10
HTTP Request Cookies extracted: 0
Post parameters extracted: 0
HTTP response Headers extracted: 10
HTTP Response Cookies extracted: 0
Unicode: 0
Double unicode: 0
Non-ASCII representable: 0
Directory traversals: 0
Extra slashes ("//"): 0
Self-referencing paths ("./"): 0
HTTP Response Gzip packets extracted: 0
Gzip Compressed Data Processed: n/a
Gzip Decompressed Data Processed: n/a
Total packets processed: 2433
===============================================================================
SMTP Preprocessor Statistics
Total sessions : 0
Max concurrent sessions : 0
===============================================================================
dcerpc2 Preprocessor Statistics
Total sessions: 0
===============================================================================
===============================================================================
SIP Preprocessor Statistics
Total sessions: 0
===============================================================================
File Preprocessor Statistics
Total file type callbacks: 7
Total file signature callbacks: 7
Total files would saved to disk: 7
Total files saved to disk: 7
Total file data saved to disk: 47466737 bytes
Total files duplicated: 0
Total files reserving failed: 0
Total file capture min: 0
Total file capture max: 0
Total file capture memcap: 0
Total files reading failed: 0
Total file agent memcap failures: 0
Total files sent: 0
Total file data sent: 0
Total file transfer failures: 0
===============================================================================
File type stats:
Type Download (Bytes) Upload (Bytes)
GZ( 33) 2 6848054 0 0
MP3( 64) 2 37257592 0 0
JPEG( 70) 2 3360645 0 0
BMP(148) 1 446 0 0
Total 7 47466737 0 0
File signature stats:
Type Download Upload
GZ( 33) 2 0
MP3( 64) 2 0
PNG( 69) 1 0
JPEG( 70) 2 0
Total 7 0
File type verdicts:
UNKNOWN: 7
LOG: 0
STOP: 0
BLOCK: 0
REJECT: 0
PENDING: 0
STOP CAPTURE: 0
Total: 7
File signature verdicts:
UNKNOWN: 7
LOG: 0
STOP: 0
BLOCK: 0
REJECT: 0
PENDING: 0
STOP CAPTURE: 0
Total: 7
Total files processed: 10
Total files data processed: 47473897 bytes
Total files buffered: 7
Total files released: 7
Total files freed: 0
Total files captured: 7
Total files within one packet: 1
Total buffers allocated: 1452
Total buffers freed: 0
Total buffers released: 1452
Maximum file buffers used: 759
Total buffers free errors: 0
Total buffers release errors: 0
Total memcap failures: 0
Total memcap failures at reserve: 0
Total reserve failures: 0
Total file capture size min: 0
Total file capture size max: 0
Total capture max before reserve: 0
Total file signature max: 0
Maximum buffers can allocate: 3196
Number of buffers in use: 0
Number of buffers in free list: 1744
Number of buffers in release list: 1452
===============================================================================
Snort exiting
This time the captured files have changed:
# ls -lS
-rw------- 1 root root 24211979 May 8 21:20
8452B621DC334D1FD44470A80540CBEF2F6869AF851B9E8C684EF9402016F692
-rw------- 1 root root 13045613 May 8 21:20
5CF142947C2957EE648457A91B69FB82F088F31205030F9A77B2AD827228C6E9
-rw------- 1 root root 6352738 May 8 21:20
DB57C532919D9ABABAC127F29DBDC05ED832394880E46CAD81A5DDE713CCB4BE
-rw------- 1 root root 2936119 May 8 21:20
B4127F43A3F455523B81179CC11AA4F28FC27F4C041D20E28AA08A32D85CB757
-rw------- 1 root root 495316 May 8 21:20
A294AA3D01CD8902BF842D320E7F2C043AF9EAD95D0E7198C3B71A0DBC9D253C
-rw------- 1 root root 424526 May 8 21:20
8863DB1EC4B02D5BCC1FB4BD03D220F7458136342CDD47CE507A5B886C6BB56C
-rw------- 1 root root 446 May 8 21:20 8D490C71A27631CF6A476F68C40965
5CB63BF32C17846A3C3C125A79046DB2C1
But they are still different from the original ones:
# ls -lS
-rw-r--r-- 1 root root 1044381696 Feb 18 20:12
ubuntu-14.04.2-desktop-amd64.iso
-rw-r--r-- 1 root root 375187792 May 8 21:07 VMware-viclient.exe
-rw-r--r-- 1 root root 101688487 Jul 10 2014 oversize_pdf_test_0.pdf
-rw-r--r-- 1 root root 14955972 May 8 21:07 MakeUp.mov
-rw-r--r-- 1 root root 6094376 May 8 21:07 video1.avi
-rw-r--r-- 1 root root 2187725 May 8 21:07 Fighter.mpg
-rw-r--r-- 1 root root 446 Mar 22 2013 tux-sw.bmp
##########################################################################
##########################################################################
Exit stats when listening from interface and type_id enabled:
===============================================================================
Run time for packet processing was 108.388974 seconds
Snort processed 256250 packets.
Snort ran for 0 days 0 hours 1 minutes 48 seconds
Pkts/min: 256250
Pkts/sec: 2372
===============================================================================
Memory usage summary:
Total non-mmapped bytes (arena): 10100736
Bytes in mapped regions (hblkhd): 122081280
Total allocated space (uordblks): 8073952
Total free space (fordblks): 2026784
Topmost releasable block (keepcost): 108544
===============================================================================
Packet I/O Totals:
Received: 256250
Analyzed: 256250 (100.000%)
Dropped: 0 ( 0.000%)
Filtered: 0 ( 0.000%)
Outstanding: 0 ( 0.000%)
Injected: 0
===============================================================================
Breakdown by protocol (includes rebuilt packets):
Eth: 256255 (100.000%)
VLAN: 0 ( 0.000%)
IP4: 256130 ( 99.951%)
Frag: 0 ( 0.000%)
ICMP: 0 ( 0.000%)
UDP: 24 ( 0.009%)
TCP: 132229 ( 51.601%)
IP6: 0 ( 0.000%)
IP6 Ext: 0 ( 0.000%)
IP6 Opts: 0 ( 0.000%)
Frag6: 0 ( 0.000%)
ICMP6: 0 ( 0.000%)
UDP6: 0 ( 0.000%)
TCP6: 0 ( 0.000%)
Teredo: 0 ( 0.000%)
ICMP-IP: 0 ( 0.000%)
IP4/IP4: 0 ( 0.000%)
IP4/IP6: 0 ( 0.000%)
IP6/IP4: 0 ( 0.000%)
IP6/IP6: 0 ( 0.000%)
GRE: 0 ( 0.000%)
GRE Eth: 0 ( 0.000%)
GRE VLAN: 0 ( 0.000%)
GRE IP4: 0 ( 0.000%)
GRE IP6: 0 ( 0.000%)
GRE IP6 Ext: 0 ( 0.000%)
GRE PPTP: 0 ( 0.000%)
GRE ARP: 0 ( 0.000%)
GRE IPX: 0 ( 0.000%)
GRE Loop: 0 ( 0.000%)
MPLS: 0 ( 0.000%)
ARP: 125 ( 0.049%)
IPX: 0 ( 0.000%)
Eth Loop: 0 ( 0.000%)
Eth Disc: 0 ( 0.000%)
IP4 Disc: 123866 ( 48.337%)
IP6 Disc: 0 ( 0.000%)
TCP Disc: 0 ( 0.000%)
UDP Disc: 0 ( 0.000%)
ICMP Disc: 0 ( 0.000%)
All Discard: 123866 ( 48.337%)
Other: 11 ( 0.004%)
Bad Chk Sum: 362 ( 0.141%)
Bad TTL: 0 ( 0.000%)
S5 G 1: 2 ( 0.001%)
S5 G 2: 3 ( 0.001%)
Total: 256255
===============================================================================
Action Stats:
Alerts: 0 ( 0.000%)
Logged: 0 ( 0.000%)
Passed: 0 ( 0.000%)
Limits:
Match: 0
Queue: 0
Log: 0
Event: 0
Alert: 0
Verdicts:
Allow: 228770 ( 89.276%)
Block: 0 ( 0.000%)
Replace: 0 ( 0.000%)
Whitelist: 27480 ( 10.724%)
Blacklist: 0 ( 0.000%)
Ignore: 0 ( 0.000%)
===============================================================================
Frag3 statistics:
Total Fragments: 0
Frags Reassembled: 0
Discards: 0
Memory Faults: 0
Timeouts: 0
Overlaps: 0
Anomalies: 0
Alerts: 0
Drops: 0
FragTrackers Added: 0
FragTrackers Dumped: 0
FragTrackers Auto Freed: 0
Frag Nodes Inserted: 0
Frag Nodes Deleted: 0
===============================================================================
===============================================================================
Stream statistics:
Total sessions: 20
TCP sessions: 14
UDP sessions: 6
ICMP sessions: 0
IP sessions: 0
TCP Prunes: 0
UDP Prunes: 0
ICMP Prunes: 0
IP Prunes: 0
TCP StreamTrackers Created: 14
TCP StreamTrackers Deleted: 14
TCP Timeouts: 0
TCP Overlaps: 0
TCP Segments Queued: 6930
TCP Segments Released: 6930
TCP Rebuilt Packets: 6331
TCP Segments Used: 6903
TCP Discards: 7
TCP Gaps: 6570
UDP Sessions Created: 6
UDP Sessions Deleted: 6
UDP Timeouts: 0
UDP Discards: 0
Events: 16
Internal Events: 0
TCP Port Filter
Filtered: 0
Inspected: 0
Tracked: 131874
UDP Port Filter
Filtered: 0
Inspected: 0
Tracked: 6
===============================================================================
HTTP Inspect - encodings (Note: stream-reassembled packets included):
POST methods: 0
GET methods: 0
HTTP Request Headers extracted: 0
HTTP Request Cookies extracted: 0
Post parameters extracted: 0
HTTP response Headers extracted: 2
HTTP Response Cookies extracted: 0
Unicode: 0
Double unicode: 0
Non-ASCII representable: 0
Directory traversals: 0
Extra slashes ("//"): 0
Self-referencing paths ("./"): 0
HTTP Response Gzip packets extracted: 0
Gzip Compressed Data Processed: n/a
Gzip Decompressed Data Processed: n/a
Total packets processed: 13165
===============================================================================
SMTP Preprocessor Statistics
Total sessions : 0
Max concurrent sessions : 0
===============================================================================
dcerpc2 Preprocessor Statistics
Total sessions: 0
===============================================================================
SSL Preprocessor:
SSL packets decoded: 68
Client Hello: 0
Server Hello: 2
Certificate: 2
Server Done: 3
Client Key Exchange: 0
Server Key Exchange: 0
Change Cipher: 2
Finished: 0
Client Application: 0
Server Application: 1
Alert: 0
Unrecognized records: 64
Completed handshakes: 0
Bad handshakes: 0
Sessions ignored: 1
Detection disabled: 1
===============================================================================
SIP Preprocessor Statistics
Total sessions: 0
===============================================================================
File Preprocessor Statistics
Total file type callbacks: 2
Total file signature callbacks: 1
Total files would saved to disk: 1
Total files saved to disk: 1
Total file data saved to disk: 446 bytes
Total files duplicated: 0
Total files reserving failed: 0
Total file capture min: 0
Total file capture max: 0
Total file capture memcap: 0
Total files reading failed: 0
Total file agent memcap failures: 0
Total files sent: 0
Total file data sent: 0
Total file transfer failures: 0
===============================================================================
File type stats:
Type Download (Bytes) Upload (Bytes)
BMP(148) 1 446 0 0
PDF(288) 1 3057259 0 0
Total 2 3057705 0 0
File signature stats:
Type Download Upload
BMP(148) 1 0
Total 1 0
File type verdicts:
UNKNOWN: 2
LOG: 0
STOP: 0
BLOCK: 0
REJECT: 0
PENDING: 0
STOP CAPTURE: 0
Total: 2
File signature verdicts:
UNKNOWN: 1
LOG: 0
STOP: 0
BLOCK: 0
REJECT: 0
PENDING: 0
STOP CAPTURE: 0
Total: 1
Total files processed: 2
Total files data processed: 3057705 bytes
Total files buffered: 2
Total files released: 1
Total files freed: 1
Total files captured: 1
Total files within one packet: 1
Total buffers allocated: 95
Total buffers freed: 94
Total buffers released: 1
Maximum file buffers used: 94
Total buffers free errors: 0
Total buffers release errors: 0
Total memcap failures: 0
Total memcap failures at reserve: 0
Total reserve failures: 0
Total file capture size min: 0
Total file capture size max: 0
Total capture max before reserve: 0
Total file signature max: 0
Maximum buffers can allocate: 3196
Number of buffers in use: 0
Number of buffers in free list: 3195
Number of buffers in release list: 1
===============================================================================
Snort exiting
This time the captured files haven't changed:
# ls -lS
-rw------- 1 root root 446 May 8 21:33
8D490C71A27631CF6A476F68C409655CB63BF32C17846A3C3C125A79046DB2C1
Best Regards,
Pablo Cantos
redborder.org / pcantos () redborder org
2015-05-08 21:49 GMT+02:00 Hui Cao (huica) <huica () cisco com>:
Hi Pablo, When listening from interfaces, you have lots of discards. Because file processing relies on data that are reassembled correctly, it won’t be called for those sessions that miss file data. In the case of PCAP, no sure why file type is not identified. It is interesting to see 47M file data for only 3326 packets. That is 24K per packet. I guess in this case, it will always hit PAF_MAX for each packet which might set each packet as single PDU(file). Can you try this setting? config paf_max: 60000 Best, Hui. From: Pablo Cantos Polaino <pcantos () redborder org> Date: Friday, May 8, 2015 at 3:29 PM To: Hui Cao <huica () cisco com> Cc: "snort-users () lists sourceforge net" <snort-users () lists sourceforge netSubject: Re: [Snort-users] File preprocessor fails to capture files IP4 Disc: 122145 ( 49.331%) IP6 Disc: 0 ( 0.000%) TCP Disc: 0 ( 0.000%) UDP Disc: 0 ( 0.000%) ICMP Disc: 0 ( 0.000%) All Discard: 122145 ( 49.331%) TCP Segments Used: 6919 TCP Discards: 48 TCP Gaps: 6459
------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users Please visit http://blog.snort.org to stay current on all the latest Snort news!
Current thread:
- File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui Cao (huica) (May 08)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui cao (May 08)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui Cao (huica) (May 08)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui Cao (huica) (May 08)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 15)
- Re: File preprocessor fails to capture files Russ (May 15)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 16)
- Re: File preprocessor fails to capture files Russ (May 17)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 18)
- Re: File preprocessor fails to capture files Pablo Cantos Polaino (May 08)
- Re: File preprocessor fails to capture files Hui Cao (huica) (May 08)