Re: RFC: a loadable plugin framework for external capture backends

[Robin Jarry via tcpdump-workers <tcpdump-workers () lists tcpdump org>]

> --- Begin Message ---
>
>
> From: "Robin Jarry" <rjarry () redhat com>
>
> Date: Sat, 17 Oct 2026 12:36:55 +0200
>
> Hi Denis,
>
> Denis Ovsienko, Oct 13, 2026 at 21:12:
> > The receiver can be slower than the sender for any reason and for any 
> > length of time.  This problem is not new in libpcap and can use the 
> > same approach as with regular network interfaces: absorb short bursts 
> > in a buffer, on a full buffer drop captured packets and increment the 
> > drop counter.  Report the statistics accurately and let the end user 
> > decide what to do about any drops.
>
> Of course, if the receiver is slower than the producer, the producer 
> should never block.
>
> But even if a write/send to a socket does not block (e.g. O_NONBLOCK) it 
> still requires a *system call*. That means context switching to kernel 
> space and absolutely no guarantee of when the kernel will agree to 
> resume execution back. For high performance datapaths, this is not 
> acceptable. We must have deterministic latency.
>
> This is my main argument for having a shared ring buffer part of the 
> plugin API from day one. It should be optional and negotiated when 
> libpcap connects to the capture backend. If the backend does not support 
> it, it should fall back to multiplexing packets and control messages in 
> the socket.
>
> > The plug-in protocol would indeed need to be versioned.  The very 
> > first version would need to include what is essential only.  Guy is 
> > most familiar with rpcap encoding, perhaps he could comment which 
> > design elements could be reused and which should be avoided?
>
> I have taken the liberty to explore the rpcap protocol and came up with 
> this draft so we can discuss on concrete proposals:
>
> # libpcap Third-Party Capture Backend Plugin
>
> A vendor-neutral protocol letting any application (DPDK datapath, bypass 
> stack, etc.) expose live capture to libpcap consumers via a lockless 
> shared-memory ring, with socket-streaming fallback. Usage: `tcpdump -i 
> <backend>:<interface>`.
>
> ## Transport tiers (negotiated, best-first)
>
> 1. Shared ring + fd passing (UN*X): AF_UNIX SEQPACKET socket at 
>   `/run/libpcap/<backend>.sock`; ring memfd passed via `SCM_RIGHTS`. 
>   Zero producer syscalls.
> 2. Shared ring, named (Windows / portable): named mapping (file mapping, 
>   `shm_open`) whose name is sent in the reply. Zero producer syscalls.
> 3. Stream fallback (universal baseline): control + length-prefixed 
>   packet records multiplexed over one local socket (AF_UNIX or loopback 
>   TCP).
>
> Consumer advertises capabilities in `HELLO`; backend selects the highest 
> tier both support. Tier 3 guarantees the API works on every platform.
>
> ## Idle / wakeup policy (negotiated)
>
> - Tiers 1-2: backend *may* pass an eventfd (UN*X, via `SCM_RIGHTS`) or 
>  OS wait handle (Windows); producer signals after a batch, consumer 
>  blocks on it and exposes it via `get_selectable_fd`.
> - No wakeup support (high-perf DPDK): consumer falls back to **adaptive 
>  backoff** (spin -> CPU-relax -> short sleep). Producer stays 
>  syscall-free.
> - Tier 3: always uses the socket fd (normal blocking recv).
>
> ## Control protocol
>
> - Packed, fixed-width structs, native byte order. Common header on every 
>  message: `{ uint16 version; uint16 type; uint32 length; }`. `version` 
>  gates all format changes; negotiated down to the lowest common value 
>  in `HELLO`. A byte-order magic in `HELLO` rejects mismatched peers.
> - Messages: `HELLO`/`HELLO_RESP` (version, capabilities, wakeup 
>  support), `LIST`/`LIST_RESP` (interface enumeration for 
>  `findalldevs`), `START`/`START_RESP` (begin capture; reply carries 
>  ring fd/name or selects stream mode, optional wakeup handle, 
>  negotiated geometry + linktype), `SET_FILTER`, `STOP`. Stream mode 
>  adds a minimal `PACKET` record.
>
> ## Negotiable sizing
>
> `slot_size`, `slot_count` (pow2), and `snap_len` are requested in 
> `START` and confirmed (possibly adjusted) in `START_RESP`. No hardcoded 
> dimensions on the wire. Backend clamps all values to its own maxima.
>
> ## Filtering
>
> BPF filter sent in `START`/`SET_FILTER`. Backend applies it **in 
> priority** if it advertises filter support (reduces ring pressure); 
> otherwise the consumer applies it locally. Untrusted filters must be 
> verified before entering the datapath (see Security).
>
> ## Shared ring (MPSC, lockless, single consumer)
>
> - Control block (shared, producer-written, *consumer read-only*): 
>  `magic`, `version`, `slot_count`, `slot_size`, `snap_len`, interface 
>  table, cache-line-isolated `prod_head` (atomic). No consumer index in 
>  shared memory.
> - Slots: `_Atomic uint32 sequence` + metadata (`pkt_len`, `cap_len`, 
>  `iface_id`, `direction`, `timestamp_ns`) + `data[]`.
> - Consumer cursor (`cons_head`) and *drop counter* live in 
>  consumer-private memory only.
>
> ### Producers (many threads, never block)
>
> `base = atomic_fetch_add(prod_head, N, relaxed)` reserves N slots per 
> burst. For each slot: write metadata, copy `cap_len` bytes, publish with 
> `store-release(sequence = pos + 1)`. Ring full -> overwrite 
> unconditionally. Producers *read only their own atomics*; they never 
> read any consumer-controlled memory.
>
> ### Consumer (single reader)
>
> Seqlock dequeue: `load-acquire(sequence)`; if `== cons_head + 1` the 
> slot is current (acquire/release pairs so all writes are visible) -> 
> snapshot-copy -> re-check `sequence` (discard on change) -> advance 
> private `cons_head`. If `sequence` jumped past `cons_head + 1`, 
> producers lapped: count the gap as dropped packets (`ps_drop`) and skip 
> `cons_head` forward using `prod_head` and `slot_count`.
>
> *Drops are counted consumer-side via sequence gaps* — exact, requiring 
> no producer knowledge of the consumer.
>
> ## Security (minimize producer attack surface)
>
> The producer (datapath) is the asset; consumers are untrusted. 
> A malicious consumer must never corrupt, stall, or crash the producer.
>
> - Consumer mapping is read-only; the passed fd is write-sealed (`memfd` 
>  + `F_SEAL_WRITE|SHRINK|GROW|SEAL`) / read-only ACL on named mappings.
> - Producer reads no consumer-controlled memory. `cons_head` is 
>  consumer-private; the ring carries no producer-read, consumer-writable 
>  field. A corrupt consumer can at worst misread already-captured 
>  packets (self-harm).
> - Control channel hardening: clamp all negotiated sizes/counts before 
>  allocating (fail-closed); verify BPF filters before datapath install 
>  (or force consumer-side filtering for untrusted peers); handle control 
>  on a dedicated thread off the datapath; per-session resource caps 
>  + idle timeout; authenticate peers (`SO_PEERCRED`/`getpeereid`); 
>  restrictive socket permissions (no world access).
> - Residual risk: consumer shares physical pages holding captured packets 
>  it is already authorized to see; no other producer memory is mapped.
>
> ## libpcap module responsibilities
>
> - Route `backend:iface` device strings; enumerate `/run/libpcap/*.sock` 
>  (or platform equivalent) for `findalldevs`.
> - `activate`: connect, HELLO, START, pick transport + idle policy, map 
>  ring or set up stream recv.
> - `read`: shm dequeue (block on wakeup handle if negotiated, else 
>  adaptive-backoff poll); stream `recv` loop.
> - `stats`: surface consumer-side drop count and produced count.
> - `get_selectable_fd`: socket fd (stream), wakeup fd (shm + wakeup), or 
>  `PCAP_ERROR` (shm + adaptive backoff).
>
> ## Versioning
>
> Control header `version` mandatory on every message, negotiated in 
> `HELLO`. A separate `version` in the ring control block governs 
> shared-memory layout independently.
>
> --
> Robin
>
> # For office use only.
>
>
> --- End Message ---
_______________________________________________
tcpdump-workers mailing list -- tcpdump-workers () lists tcpdump org
To unsubscribe send an email to tcpdump-workers-leave () lists tcpdump org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s