Re: RFC: a loadable plugin framework for external capture backends

[Michael Richardson <mcr () sandelman ca>]

Robin Jarry via tcpdump-workers <tcpdump-workers () lists tcpdump org> wrote:
    > Jardin, the author of the PR, and Robin Jarry) are jointly moving the
    > discussion of a libpcap plugin framework here, to the mailing list, as

So, I want to distinguish between a **plugin** vs a stable API/ABI that DPDK,
VPP, etc. could work against.  Or a stable ABI that libpcap could work against.

    > Modern userspace dataplanes (DPDK-based routers such as grout, OVS-DPDK, VPP,
    > and similar) move packets entirely in userspace. Their interfaces are
    > invisible to the kernel, so `tcpdump -i <iface>` cannot open them as a live
    > source. Capturing traffic from a running dataplane today means either a
    > bespoke CLI tool per project, or building capture support directly into
    > libpcap.

If we can create an interface which is not tied to a specific (DPDK) version,
then I prefer that the support be statically built in.

    > Existing remote-capture solutions such as rpcapd do not help here
    > either: they require the dataplane to write every captured packet to a
    > TCP/UDP socket,

The rpcapd solution should not be horrible if the percent of traffic is low,
and the BPF program runs inside of the user-space engine.
This might be the best story for "casual" captures.
It won't due for someone trying to eat 100Gb/s of live DNS traffic for
analysis.   https://adam.pages.nic.cz/dns-probe/Running.html does not, AFAIK,
use libpcap for these preformance reasons, offering TPACKET or DPDK integrations.

I think that the rpcap protocol *itself* could be better, mind you.

    > A plugin framework decouples the two. A dataplane project builds its own
    > pcap-<name>.so against its own headers, ships it with its own packages, and
    > upgrading the dataplane only means rebuilding the plugin, not libpcap. If the
    > plugin is absent or fails to load, libpcap keeps working exactly as
    > before. This is the same reason PAM, NSS, GStreamer, OpenSSL providers and
    > Mesa all converged on a loadable-module model.

Some of these are good examples, and there have been problems with all of
them over the decades :-)
NSS and PAM have had many vulnerabilities/gotchas in decades past.
(I also have many unpritable opinions about the OpenSSL provider interfaces)

So, let me suggest different interfaces which are relatively stable, and are
already shared memory based: TPACKET_V3 and AF_XDP:
  https://www.kernel.org/doc/html/latest/networking/af_xdp.html

I recognize that these are two different but very similiar things, and I am
unclear to what extent the Linux kernel treats them in similiar code paths.
I believe that VPP, DPDK, etc. use AF_XDP to get packets from the kernel,
and send them back to the device.  (I think that there are also still vendor
proprietary paths?)

There is a win here in that libpcap could wind up with an AF_XDP capture
path.  Obviously, the setsockopt() calls that AF_XDP requires have to
replaced with a different communication.  Probably a UNIX domain socket,
likely the shared memory segments used for the packets and rings would be
passed by fd through that socket.   (Which way?  Probably both)

I think, but I can not be certain, that this might permit zero-copy loans of
the packet buffers from VPP/DPDK/etc. to libpcap.
Read-only?  Copy-on-Write maybe useful of the engine needs to mutate the data
while libpcap is still processing... but likely that's too expensive.

    > Worth noting about this use case, because it bears directly on the security
    > discussion below: the plugin needs *no elevated privileges at all*. The only
    > resource it touches is a UNIX socket, over which it asks the grout daemon to
    > start/stop a capture. Packets are then delivered through an ephemeral shared
    > memory segment that the daemon owns. libpcap itself opens nothing privileged
    > for this backend.

Yes, but what privileges libpcap/user needs to talk to DPDK is not really the
concern.

1. some systems (distros) setuid on tcpdump.  I've never liked this, but I
   understand it.
2. some systems (distros) use capabilities, set on the tcpdump executable.
3. but, many of us prefer to just sudo.

I'd like to do more to split things up while not hurting performance too
much.   OpenBSD has done some of this, but in it's own way.  I think that
they are more concerned about the security around the dissectors, and I want
that isolation as well.  Particularly for the "sudo" case.

The unix socket with ring buffers inteface I suggest above could also
become the boundary between a front-end (filtering/printing) process, and the
backend (some priveledge required to start) capture process.
That is, libpcap could fork()/exec() or tickle some systemd thing, to get the
access it needs, and then use the same ring ABI.

The concern is that on systems with (1) and (2) that any plugin here runs
with privileges the invoker does not have.  The number of hoops that things go
through on loadable modules to (a) bolt down where they come from,
(b) prevent all symlink and other timing attacks against that path,
while (c) not pissing off developers who really really want to load and then
debug ../fubar/borg/myplugin.so.

[When I have done debugging of ruby-openssl.so module, often buried deep
under a web framework, the only practical way to be sure of what SOs are
loaded is to compile the openssl statically, and then link ruby-openssl.so
against it statically.  And then you might see how the provider mechanism is
a challenge which results in too hard to debug/reproduce code.]

    > - Discovery. Where are plugins found, and in what order? (The PR uses    a
    > single compiled-in directory plus an environment override that is
    > dropped under elevated privileges, like ld.so with LD_LIBRARY_PATH.)

Yes, and that's where I begin to get afraid, because features were added to
ld.so, and sometimes they interacted in curious and insecure ways.
There may be *ld.so* options that affect us even.

    > - Enumeration vs selection. Do plugins only contribute devices to
    > pcap_findalldevs(), or can they intercept arbitrary device names?

    > Our
    > preference: a plugin owns a namespace prefix (e.g. "grout:") and    is
    > consulted only for names in its namespace. This makes "how do    multiple
    > plugins coexist" answerable by construction.

So, the devices would not show up for --list-interfaces, but if one does -i grout:dpdp/selector
then it loads libpcap-dpdp-plugin.so, and passes "selector" to it?

The alternative is that tcpdump has a --load-plugins dpdk,vpp ... and if that
shows up before --list-interfaces, then it enumerates the devices from those
plugins?

I would prefer --list-interfaces *NOT* attempt to load all the plugins in the
default directory, and ask them for a list :-)

That would otherwise be the best thing for users.
That's mostly what I think wireshark UI users would want.
Maybe a specific executable that can do this, returning JSON, would satisfy.

Further, all of this probably has to work when it's rpcapd (under wireshark,
from a windows desktop user...) that is doing this.
That's where having multiple plugins loaded might be required.

*The rpcap client becomes a plugin*

    > - Parallelism / pre-emption. Denis asked how multiple plugins work in
    > parallel or pre-empt each other. Our proposed answer: they do not
    >interact at all. Each pcap_t is bound to exactly one backend chosen    at
    > create/activate time; plugins never see each other's handles.    Anything
    > else should be out of scope for a first version.

Yes, that's good.
I'm not finding a good reason for more than one plugin would exist per pcap_t.
I'm sure that there are programs that use more than one pcap_t, and quite
possibly they might need to have different plugins.

    > - ABI. Plugins must not see pcap-int.h. The proposal is a versioned,
    > narrow accessor API (pcap_plugin_*) so libpcap internals stay    private
    > and the plugin ABI is explicit and stable.

Agreed.
No public structures ideally, only getters/setters?

    > - Failure behaviour. A plugin that is missing, unloadable, or reports    a
    > mismatched ABI version must be a silent no-op, never a fatal error    for
    > libpcap or its callers.

I agree that it shouldn't call exit(2)!
I also agree that error strings need to be returned, not fprintf(stderr,..).
I'm not understanding the no-op :-)

    > 4. The attack surface
    > =====================

    > This is the part that most needs the group's judgement, and we do not want to
    > hand-wave it.

    > The honest position is that dlopen()ing code into a process that may hold
    > capture privileges increases the attack surface, and the framework must be
    > built so that the increase is as close to zero as possible.

Thanks.
I'd like _tcpdump_ to drop any privilege if it is told to load a plugin.

While no specific local kernel privilege is required to talk to DPDK, I would
assume that the API into things requires something... group permissions to
open a socket in a magic directory?
That kind of thing can just be an attribute of the user invoking things?

    > - Off by default. The loader should be a compile-time option, disabled    by
    > default, as fxlb already requested on the PR. Distributions and
    > dataplane projects opt in; a stock libpcap build carries none of it.

Given this, does the plugin actually need to be loadable?

The problem with the existing contribution was the lack of
maintenance/maintainers.  We needed private headers to even attempt to compile.
If we don't compile it regularly, then we won't really know if it is broken.
If the can stabilize an API (not ABI), then maybe that's the simpler way.

Alternatively: fork libpcap, rip EVERYTHING else out ("git rm", not edits).
Leave that as one or two commits with the right code, and just rebase that on
libpcap.git.   Maybe we'd even accept that as a continuing branch.

    > - No privileged loading. Since the motivating backends need no    elevated
    > privileges (see section 2), libpcap could refuse to load    plugins while
    > running with elevated privileges, or drop those    privileges first and
    > only proceed if the drop succeeded. A single    tcpdump binary could still
    > do `tcpdump -i eth0` (opens a raw socket,    then drops) and `tcpdump -i
    > grout:p0` (no privileges needed).

yes, I think everyone is in agreement here.
I don't know if that will work for wireshark or other things.

    > - Trusted location only. Plugins load only from a root-owned,
    > non-user-writable directory; the environment override is ignored    under
    > elevated privileges, exactly as ld.so treats LD_LIBRARY_PATH.

Well, if libpcap drops any privileges it might have had the moment
--load-plugin is seen (but, before acting on it), then actually, this is moot.

    > - Optional, on top: detached-signature verification against keys in    a
    > root-owned directory, for deployments that want it. Possibly    redundant
    > with distro package signing.

No.  It will frustrate users that self-compile.

    > 5. Maintenance burden
    > =====================

    > To Denis's concern that a framework nobody uses, or one whose plugins become
    > TTG's responsibility, would be an expensive mistake: we agree, and we think
    > the framework should be designed so that no plugin is ever TTG's
    > responsibility. Plugins live in and are maintained by their own projects
    > (grout being a concrete, active example). Inside libpcap the recurring cost
    > is just the loader and the accessor ABI. If we keep the ABI narrow and the
    > loader off by default, that cost stays small and self-contained, and there
    > are no repository-boundary problems inside libpcap itself.

The best way to amortize and test the cost of the ABI is for libpcap to use
that ABI to do, for instance TPACKET_V3.   This might let us remove some of
the TPACKET3 bug-work around code by having an "older" plugin.
We have lots of #ifdef HAVE_LIBNL around nl80211 code, and maybe that could
be a plugin.  Maybe we could drop TPACKET_V2, although that might not be much
of a win.

    > We are aware this will take time, and that libpcap 1.11.x is still
    > stabilising; there is no urgency on our side beyond wanting to do it
    > right. Thanks in advance for your feedback.

...

    > --
    > Vincent Jardin -- Free Mobile, Iliad
    > Robin Jarry -- Red Hat

Thank you for the engaging email!

_______________________________________________
tcpdump-workers mailing list -- tcpdump-workers () lists tcpdump org
To unsubscribe send an email to tcpdump-workers-leave () lists tcpdump org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s