Re: RFC: a loadable plugin framework for external capture backends

[Vadim Goncharov <vadimnuclight () gmail com>]

On Sat, 17 Oct 2026 12:36:55 +0200
Robin Jarry via tcpdump-workers <tcpdump-workers () lists tcpdump org> wrote:

> Hi Denis,
>
> Denis Ovsienko, Oct 13, 2026 at 21:12:
> > The receiver can be slower than the sender for any reason and for any 
> > length of time.  This problem is not new in libpcap and can use the 
> > same approach as with regular network interfaces: absorb short bursts 
> > in a buffer, on a full buffer drop captured packets and increment the 
> > drop counter.  Report the statistics accurately and let the end user 
> > decide what to do about any drops.  
>
> Of course, if the receiver is slower than the producer, the producer 
> should never block.
>
> But even if a write/send to a socket does not block (e.g. O_NONBLOCK) it 
> still requires a *system call*. That means context switching to kernel 
> space and absolutely no guarantee of when the kernel will agree to 
> resume execution back. For high performance datapaths, this is not 
> acceptable. We must have deterministic latency.
>
> This is my main argument for having a shared ring buffer part of the 
> plugin API from day one. It should be optional and negotiated when 
> libpcap connects to the capture backend. If the backend does not support 
> it, it should fall back to multiplexing packets and control messages in 
> the socket.
>
> > The plug-in protocol would indeed need to be versioned.  The very 
> > first version would need to include what is essential only.  Guy is 
> > most familiar with rpcap encoding, perhaps he could comment which 
> > design elements could be reused and which should be avoided?  
>
> I have taken the liberty to explore the rpcap protocol and came up with 
> this draft so we can discuss on concrete proposals:
>
> # libpcap Third-Party Capture Backend Plugin
>
> A vendor-neutral protocol letting any application (DPDK datapath, bypass 
> stack, etc.) expose live capture to libpcap consumers via a lockless 
> shared-memory ring, with socket-streaming fallback. Usage: `tcpdump -i 
> <backend>:<interface>`.
>
> ## Transport tiers (negotiated, best-first)
>
> 1. Shared ring + fd passing (UN*X): AF_UNIX SEQPACKET socket at 
>    `/run/libpcap/<backend>.sock`; ring memfd passed via `SCM_RIGHTS`. 

/var/run is more portable (e.g. BSDs)

>    Zero producer syscalls.
> 2. Shared ring, named (Windows / portable): named mapping (file mapping, 
>    `shm_open`) whose name is sent in the reply. Zero producer syscalls.
> 3. Stream fallback (universal baseline): control + length-prefixed 
>    packet records multiplexed over one local socket (AF_UNIX or loopback 
>    TCP).
>
> Consumer advertises capabilities in `HELLO`; backend selects the highest 
> tier both support. Tier 3 guarantees the API works on every platform.
>
> ## Idle / wakeup policy (negotiated)
>
> - Tiers 1-2: backend *may* pass an eventfd (UN*X, via `SCM_RIGHTS`) or 
>   OS wait handle (Windows); producer signals after a batch, consumer 
>   blocks on it and exposes it via `get_selectable_fd`.
> - No wakeup support (high-perf DPDK): consumer falls back to **adaptive 
>   backoff** (spin -> CPU-relax -> short sleep). Producer stays 
>   syscall-free.
> - Tier 3: always uses the socket fd (normal blocking recv).
>
> ## Control protocol
>
> - Packed, fixed-width structs, native byte order. Common header on every 
>   message: `{ uint16 version; uint16 type; uint32 length; }`. `version` 

Why version in every message?

>   gates all format changes; negotiated down to the lowest common value 
>   in `HELLO`. A byte-order magic in `HELLO` rejects mismatched peers.
> - Messages: `HELLO`/`HELLO_RESP` (version, capabilities, wakeup 
>   support), `LIST`/`LIST_RESP` (interface enumeration for 
>   `findalldevs`), `START`/`START_RESP` (begin capture; reply carries 
>   ring fd/name or selects stream mode, optional wakeup handle, 
>   negotiated geometry + linktype), `SET_FILTER`, `STOP`. Stream mode 
>   adds a minimal `PACKET` record.

Format of other messages is not described...

> ## Negotiable sizing
>
> `slot_size`, `slot_count` (pow2), and `snap_len` are requested in 
> `START` and confirmed (possibly adjusted) in `START_RESP`. No hardcoded 
> dimensions on the wire. Backend clamps all values to its own maxima.
>
> ## Filtering
>
> BPF filter sent in `START`/`SET_FILTER`. Backend applies it **in 
> priority** if it advertises filter support (reduces ring pressure); 
> otherwise the consumer applies it locally. Untrusted filters must be 
> verified before entering the datapath (see Security).
>
> ## Shared ring (MPSC, lockless, single consumer)
>
> - Control block (shared, producer-written, *consumer read-only*): 
>   `magic`, `version`, `slot_count`, `slot_size`, `snap_len`, interface 
>   table, cache-line-isolated `prod_head` (atomic). No consumer index in 
>   shared memory.
> - Slots: `_Atomic uint32 sequence` + metadata (`pkt_len`, `cap_len`, 
>   `iface_id`, `direction`, `timestamp_ns`) + `data[]`.
> - Consumer cursor (`cons_head`) and *drop counter* live in 
>   consumer-private memory only.
>
> ### Producers (many threads, never block)
>
> `base = atomic_fetch_add(prod_head, N, relaxed)` reserves N slots per 
> burst. For each slot: write metadata, copy `cap_len` bytes, publish with 
> `store-release(sequence = pos + 1)`. Ring full -> overwrite 
> unconditionally. Producers *read only their own atomics*; they never 
> read any consumer-controlled memory.
>
> ### Consumer (single reader)
>
> Seqlock dequeue: `load-acquire(sequence)`; if `== cons_head + 1` the 
> slot is current (acquire/release pairs so all writes are visible) -> 
> snapshot-copy -> re-check `sequence` (discard on change) -> advance 
> private `cons_head`. If `sequence` jumped past `cons_head + 1`, 
> producers lapped: count the gap as dropped packets (`ps_drop`) and skip 
> `cons_head` forward using `prod_head` and `slot_count`.
>
> *Drops are counted consumer-side via sequence gaps* — exact, requiring 
> no producer knowledge of the consumer.
>
> ## Security (minimize producer attack surface)
>
> The producer (datapath) is the asset; consumers are untrusted. 
> A malicious consumer must never corrupt, stall, or crash the producer.
>
> - Consumer mapping is read-only; the passed fd is write-sealed (`memfd` 
>   + `F_SEAL_WRITE|SHRINK|GROW|SEAL`) / read-only ACL on named mappings.

What is this "sealing" ? And I do not see memfd_create() on OpenBSD; in
FreeBSD function is briefly described as:

     The memfd_create() function creates an anonymous shared memory object,
     identical to that created by shm_open() when SHM_ANON is specified.

> - Producer reads no consumer-controlled memory. `cons_head` is 
>   consumer-private; the ring carries no producer-read, consumer-writable 
>   field. A corrupt consumer can at worst misread already-captured 
>   packets (self-harm).
> - Control channel hardening: clamp all negotiated sizes/counts before 
>   allocating (fail-closed); verify BPF filters before datapath install 
>   (or force consumer-side filtering for untrusted peers); handle control 
>   on a dedicated thread off the datapath; per-session resource caps 
>   + idle timeout; authenticate peers (`SO_PEERCRED`/`getpeereid`); 
>   restrictive socket permissions (no world access).
> - Residual risk: consumer shares physical pages holding captured packets 
>   it is already authorized to see; no other producer memory is mapped.
>
> ## libpcap module responsibilities
>
> - Route `backend:iface` device strings; enumerate `/run/libpcap/*.sock` 
>   (or platform equivalent) for `findalldevs`.
> - `activate`: connect, HELLO, START, pick transport + idle policy, map 
>   ring or set up stream recv.
> - `read`: shm dequeue (block on wakeup handle if negotiated, else 
>   adaptive-backoff poll); stream `recv` loop.
> - `stats`: surface consumer-side drop count and produced count.
> - `get_selectable_fd`: socket fd (stream), wakeup fd (shm + wakeup), or 
>   `PCAP_ERROR` (shm + adaptive backoff).
>
> ## Versioning
>
> Control header `version` mandatory on every message, negotiated in

Again, why every?
 
> `HELLO`. A separate `version` in the ring control block governs 
> shared-memory layout independently.
 
That's acceptable.

-- 
WBR, @nuclight
_______________________________________________
tcpdump-workers mailing list -- tcpdump-workers () lists tcpdump org
To unsubscribe send an email to tcpdump-workers-leave () lists tcpdump org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s