Re: exit status codes semantics in tests

[Denis Ovsienko <denis () ovsienko info>]

On Fri, 16 Oct 2026 10:36:49 -0700
Guy Harris <gharris () sonic net> wrote:

> On Oct 16, 2026, at 4:59 AM, Denis Ovsienko <denis () ovsienko info>
> wrote:
>
> > The current use/non-use of EX_DATAERR (65) and EX_OK (0) in
> > filtertest, translatetest and enumeratetest, as well as of
> > EX_NOINPUT (66) in filtertest, has to remain intact because that's
> > how TESTrun tells whether a test passed or failed, e.g. filtertest
> > usage message says:
> >
> > Exit status codes:
> >    0: All input has been successfully processed.
> >   65: libpcap has orderly rejected the filter.
> >   66: libpcap has orderly rejected the savefile.
> >   70: Unexpected libpcap error.
> >   71: Unexpected OS error.
> >   74: Unexpected I/O or file format error.
> >   64: This executable has been invoked incorrectly.  
>
> So the problem is, at least in part, that the BSD <sysexits.h> header
> (also adopted by at least some C libraries used in Linux
> distributions) is that, as I remember, it started out as error exits
> for sendmail (or maybe even delivermail!)), and thus doesn't include
> all the exit codes that we might want for our tests.
>
> This means that an attempt to have all the test exit codes "make
> sense" may fail because it's impossible to succeed.
>
> However, these test programs aren't commands for the user to run in
> arbitrary scripts; they're only intended to be run manually by a
> libpcap developer when testing a change, or by a libpcap test script,
> so strictly "making sense" isn't necessarily - all that's necessary
> is that the test scripts can do the tests correctly.

Yes, pretty much.  I used something that at the time I thought was 100%
commonplace.  The set of values could be mapped 1:1 to another, purely
internal set, and the build-time check for sysexits.h could be dropped.
Meanwhile implementing the current revision has made some silent bugs
manifest at least.

[...]

> > EX_USAGE concerns only the composition of command-line arguments
> > (which leads me to notice that parsing of the DLT does not conform
> > to this rule), so if the use case demands a filter expression, but
> > it is not given as an /opaque/ argument, filtertest exits with
> > EX_USAGE to indicate that it is impossible to produce the test
> > result.  The root cause is in TESTrun.  
>
> I.e., the intent is that it means that the test failed because
> there's a bug in the test where it's failing due to an error with the
> command-line arguments, but either it's not supposed to fail or it
> *is* supposed to fail because it's testing to make sure libpcap
> catches some error in, for example, a filter, but it's failing for
> some *other* command-line-argument reason.

If TESTrun has received EX_USAGE from a test program, it means a bug in
TESTrun because every command line is an internal product of TESTrun.

> A problem with filter expressions here is that it could fail for some
> reason *other* than the reason we intended. The only way to know
> whether the test fails for the right reason would be to check the
> error message; if the test doesn't do this itself, it can just exit
> with some particular exit code and leave it to the test script to
> compare the error message with the expected error message. It doesn't
> really matter, in that case, whether EX_USAGE or EX_DATAERR is used -
> "the test succeeded because the test program failed for the right
> reason" vs. "the test failed because the test program failed for some
> *other* reason" would be resolved by checking the error message
> against the expected error message.

The current implementation of filter reject tests is "belt and braces":
both EX_DATAERR to prove that the code path has successfully reached
pcap_compile() and a substring match in the resulting standard error,
as discussed above.  That's because the error substring was a later
addition in this group of tests.

There is at least one other context: when pcap_compile() is not in the
code path (the filter program is a cbpf-savefile), a failure to
validate the program is EX_DATAERR instead of the usual EX_SOFTWARE
because the contents of cbpf-savefile is the opaque test data, not an
output of pcap_compile().

filtertest code over time grew convoluted.  Perhaps in future some use
scenarios and associated code paths could become separate functions or
programs.  It would require some careful planning and prototyping, and
at the time I am trying to get other work done.

> Another case would be if we had a program that reads a capture file
> and prints out the packet header and data for each packet (i.e., it
> prints the time stamp either as raw seconds.microseconds or as a UTC
> time stamp, the captured length and the on-the-network length, and a
> hex dump of the packet data). This test might be *intended* to
> "fail", in the sense that the file is testing whether the libpcap
> file-reading code checks for invalid capture files.

Time stamps in filter apply tests have been discussed once or twice
before, this could use a command-line option.  Other packet meta-data
could use the same or other options.  To make complete sense, this
would need meaningful savefiles and meaningful test definitions to
exercise specific code paths.

For packet data it is not clear what code path it would test.

> As I read sysexits.h, EX_DATAERR looks as if it covers this case. The
> script would presumably treat that as a "so far, so good" error and
> then check to make sure it fails with the *expected* error message,
> which is a test *success*, or with some *other* error message, which
> is a test *failure*.

Does not look wrong at a glance, but running code could prove the method
would need to be different.  It would help to state what problem needs
to be solved.

> > Alternatively, it can exit with, e.g. EX_IOERR to indicate that the
> > specified filter expression file could not be read, thus it is
> > impossible to produce the test result.  The root cause is most
> > likely in the filesystem.  
>
> As I read EX_IOERR, that's intended for a read or write operation
> failing; for this particular test, you're presumably referring to a
> case where the filter expression is read from the file. The
> particular *reason* for the failure doesn't matter, as a failure
> isn't a problem in libpcap or with the test program, it's a problem
> with the system running the tests.

Yes, filter expressions can and in the TESTrun case do come from a
temporary file.  What matters is that sometimes the result of a test
must be conveyed as "could not run the test at all", not "test passed"
or "test failed".

> > Otherwise, it can exit with EX_DATAERR to indicate that the opaque
> > piece of data is present and libpcap works and the result is that
> > libpcap rejects the data.  The root cause is in the filter
> > expression and any other test parameters.  Whether this is the
> > expected result for the test is up to the test type and definition
> > in TESTrun.  
>
> To which opaque piece of data are you referring?  It sounds as if
> you're referring to reading the filter expression from a file, so
> that's expected to be a libpcap filter express (presumably a large
> one that would be impossible or undesirable to put on a command
> line). I wouldn't expect an invalid filter repression to cause
> different exit statuses for expressions on the command line and
> expressions in a file.

The filter expression string (or the filter program cbpf-savefile) is
mostly opaque to filtertest, in the sense that filtertest has no means
to interpret the meaning, and also an otherwise invalid filter can be
the correct input for a test that exercises a "does this fail to fail?"
code path.  So the meaning of a test data item is irrelevant in the
sense of EX_USAGE, only its presence.

If the test data has been successfully supplied, but the libpcap
function rejected it, the test program exits with EX_DATAERR regardless
of how the data was supplied and which libpcap function was used for
the specific test.

> > Optionally, it can exit with EX_NOINPUT to indicate that libpcap has
> > failed opening the specified savefile.  The root cause could be a
> > filesystem error or a malformed file.  
>
> Or a path name that refers to a file that doesn't exist.
>
> Either EX_DATAERR ("The input data was incorrect in some way") or
> EX_NOINPUT ("An input file (not a system file) did not exist *or was
> not readable*") could work for this.
>
> We could *also* define our own exit codes, either
> test-program-specific or a set of codes for all to use; sysexits.h
> defines EX__BASE as the "base" of the error codes it defines, so we
> could define a set of error codes from 1 to 63 for use by test
> programs.

Possibly.  I am not ready now to reason about pro et contra of merging
some status codes, it would take some time to do right if something
needs to be improved.  The current implementation does not, as far as I
am aware, fail to detect a failure of any test within the current test
coverage.  Under normal conditions the only cause of a test failure is a
timeout due to external effects.  In other words, it works, so to me it
would make more sense to fix first what has been proven broken.

-- 
    Denis Ovsienko
_______________________________________________
tcpdump-workers mailing list -- tcpdump-workers () lists tcpdump org
To unsubscribe send an email to tcpdump-workers-leave () lists tcpdump org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s