libpcap delivering the same packet more than once under high load?

[Jim Lloyd <jlloyd () silvertailsystems com>]

We use libpcap 1.1.1 on linux 2.6.18 (CentOS 5.5) to sniff http traffic. I
have a test driver application written with libcurl than uses the curl multi
API to simultaneously download (using HTTP GET requests) N jpg images from a
server running apache httpd. This combination is able to saturate a GigE
connection in the download direction, and our libpcap application is able to
reconstruct the TCP streams and parse the HTTP messages with no errors. This
is excellent (great job guys!) and we're very pleased with the performance.
But it appears that libpcap does have anomalous behavior when the GigE link
is saturated.

We track the amount of bits received from libpcap by accumulating
pcap_pkthdr.len*8 for each packet received. We also track other useful
statistics. Here are results from a number of runs:

 N      cap mbps     packets      data       dup   requests   eth mbps
 1          350       22100      6988        34        469        369
 2          752       44547     14013       364        961        757
 3          977       52289     15603      1400       1138        906
 4         1143       55223     16025      2569       1200        950
 5         1333       57891     16681      4885       1234        978
 6         1424       57834     16192      6033       1262        998
10         2149       58213     15662     12529       1271       1003
20         3282       49607     14133     17101       1272        997
30         3169       42122     13481     14953       1271        975

The columns are:
N                     number of concurrent requests
cap mbps       The Mbps as measured by accumulating pcap_pkthdr.len*8 (and
then dividing by 2^20, not 10^6).
packets           The total packets per second received from lipbcap
data                 The number of data packets per second that were used in
reconstructing the TCP stream
dup                  The number of data packets that were discarded as
duplicates while reconstructing the TCP stream
requests          The number of HTTP requests parsed from the TCP streams
eth mbps         The mbps seen on the eth NIC, computed by polling
/proc/net/dev.
                         Includes both inbound & output data.

The data shows that we effectively saturate the GigE link at around 5
concurrent connections, where we see peak or near-peak rates for most
metrics. As we increase the number of concurrent connections, the number of
requests per second stays relatively flat, as does the Mbps seen at the
ethernet NIC. But strangely, the Mbps of the data delivered by libpcap to
the application continues to climb, to up to about 3Gbps. This increase in
the data rate seems to be due to the continuing increase in the number of
duplicate packets per second.

These duplicate packets cannot be unique packets that were retransmitted
between the two machines on the layer 1 GigE link, because if there was a
significant increase in retransmission duplicates on the link, they would
have competed for the fixed 1Gbps channel capacity. But the data is
consistent with libpcap delivering the same packets more than once.

Is this known behavior? It's relatively benign behavior and we don't expect
to see saturated links in any production environment, so if this is a bug in
libpcap, it's probably low priority to fix.

FYI we use pcap_dispatch, and our dispatch callback simply copies the packet
and then queues the copied packet onto one of several queues that are each
consumed by a different worker thread. The statistics for "packets" are
gathered by the main thread. The statistics for "data" and "dup" are
gathered in the worker threads. We do have cross-checks to ensure that the
total packet rate seen by the worker threads is consistent with the packet
rate seen by the main thread.

Thanks,
Jim Lloyd
Silver Tail Systems
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.