Re: exit status codes semantics in tests

[Guy Harris <gharris () sonic net>]

On Oct 16, 2026, at 4:59 AM, Denis Ovsienko <denis () ovsienko info> wrote:

> The current use/non-use of EX_DATAERR (65) and EX_OK (0) in filtertest,
> translatetest and enumeratetest, as well as of EX_NOINPUT (66) in
> filtertest, has to remain intact because that's how TESTrun tells
> whether a test passed or failed, e.g. filtertest usage message says:
>
> Exit status codes:
>    0: All input has been successfully processed.
>   65: libpcap has orderly rejected the filter.
>   66: libpcap has orderly rejected the savefile.
>   70: Unexpected libpcap error.
>   71: Unexpected OS error.
>   74: Unexpected I/O or file format error.
>   64: This executable has been invoked incorrectly.

So the problem is, at least in part, that the BSD <sysexits.h> header (also adopted by at least some C libraries used 
in Linux distributions) is that, as I remember, it started out as error exits for sendmail (or maybe even 
delivermail!)), and thus doesn't include all the exit codes that we might want for our tests.

This means that an attempt to have all the test exit codes "make sense" may fail because it's impossible to succeed.

However, these test programs aren't commands for the user to run in arbitrary scripts; they're only intended to be run 
manually by a libpcap developer when testing a change, or by a libpcap test script, so strictly "making sense" isn't 
necessarily - all that's necessary is that the test scripts can do the tests correctly.

The codes above are, according to the comment in the Xcode 27 <sysexits.h> (directly taken from BSD):

   0: All input has been successfully processed:

 *      EX_USAGE -- The command was used incorrectly, e.g., with
 *              the wrong number of arguments, a bad flag, a bad
 *              syntax in a parameter, or whatever.

  65: libpcap has orderly rejected the filter:

 *      EX_DATAERR -- The input data was incorrect in some way.
 *              This should only be used for user's data & not
 *              system files.

  66: libpcap has orderly rejected the savefile:

 *      EX_NOINPUT -- An input file (not a system file) did not
 *              exist or was not readable.  This could also include
 *              errors like "No message" to a mailer (if it cared
 *              to catch it).

  70: Unexpected libpcap error:

 *      EX_SOFTWARE -- An internal software error has been detected.
 *              This should be limited to non-operating system related
 *              errors as possible.

  71: Unexpected OS error:

 *      EX_OSERR -- An operating system error has been detected.
 *              This is intended to be used for such things as "cannot
 *              fork", "cannot create pipe", or the like.  It includes
 *              things like getuid returning a user that does not
 *              exist in the passwd file.

  74: Unexpected I/O or file format error:

 *      EX_IOERR -- An error occurred while doing I/O on some file.

  64: This executable has been invoked incorrectly:

 *      EX_USAGE -- The command was used incorrectly, e.g., with
 *              the wrong number of arguments, a bad flag, a bad
 *              syntax in a parameter, or whatever.

> Here the currently implemented test logic hinges on the first three
> codes and the last four codes just make test failure debugging easier.
> Other test programs or exit status codes have no impact on the existing
> tests.
>
> The semantic difference between EX_USAGE, EX_SOFTWARE and EX_DATAERR
> originates in filtertest as the first automated test program and
> generally implements the following logic (the description may be not
> entirely accurate, but should convey the idea), which allows to tell
> expected failures of reject tests from unexpected failures that account
> to transient host effects and bugs.
>
> EX_USAGE concerns only the composition of command-line arguments (which
> leads me to notice that parsing of the DLT does not conform to this
> rule), so if the use case demands a filter expression, but it is not
> given as an /opaque/ argument, filtertest exits with EX_USAGE to
> indicate that it is impossible to produce the test result.  The root
> cause is in TESTrun.

I.e., the intent is that it means that the test failed because there's a bug in the test where it's failing due to an 
error with the command-line arguments, but either it's not supposed to fail or it *is* supposed to fail because it's 
testing to make sure libpcap catches some error in, for example, a filter, but it's failing for some *other* 
command-line-argument reason.

A problem with filter expressions here is that it could fail for some reason *other* than the reason we intended. The 
only way to know whether the test fails for the right reason would be to check the error message; if the test doesn't 
do this itself, it can just exit with some particular exit code and leave it to the test script to compare the error 
message with the expected error message. It doesn't really matter, in that case, whether EX_USAGE or EX_DATAERR is used 
- "the test succeeded because the test program failed for the right reason" vs. "the test failed because the test 
program failed for some *other* reason" would be resolved by checking the error message against the expected error 
message.

Another case would be if we had a program that reads a capture file and prints out the packet header and data for each 
packet (i.e., it prints the time stamp either as raw seconds.microseconds or as a UTC time stamp, the captured length 
and the on-the-network length, and a hex dump of the packet data). This test might be *intended* to "fail", in the 
sense that the file is testing whether the libpcap file-reading code checks for invalid capture files.

As I read sysexits.h, EX_DATAERR looks as if it covers this case. The script would presumably treat that as a "so far, 
so good" error and then check to make sure it fails with the *expected* error message, which is a test *success*, or 
with some *other* error message, which is a test *failure*.

> Alternatively, it can exit with, e.g. EX_IOERR to indicate that the
> specified filter expression file could not be read, thus it is
> impossible to produce the test result.  The root cause is most likely in
> the filesystem.

As I read EX_IOERR, that's intended for a read or write operation failing; for this particular test, you're presumably 
referring to a case where the filter expression is read from the file. The particular *reason* for the failure doesn't 
matter, as a failure isn't a problem in libpcap or with the test program, it's a problem with the system running the 
tests.

> Otherwise, it can exit with EX_DATAERR to indicate that the opaque
> piece of data is present and libpcap works and the result is that
> libpcap rejects the data.  The root cause is in the filter expression
> and any other test parameters.  Whether this is the expected result for
> the test is up to the test type and definition in TESTrun.

To which opaque piece of data are you referring?  It sounds as if you're referring to reading the filter expression 
from a file, so that's expected to be a libpcap filter express (presumably a large one that would be impossible or 
undesirable to put on a command line). I wouldn't expect an invalid filter repression to cause different exit statuses 
for expressions on the command line and expressions in a file.

> Optionally, it can exit with EX_NOINPUT to indicate that libpcap has
> failed opening the specified savefile.  The root cause could be a
> filesystem error or a malformed file.

Or a path name that refers to a file that doesn't exist.

Either EX_DATAERR ("The input data was incorrect in some way") or EX_NOINPUT ("An input file (not a system file) did 
not exist *or was not readable*") could work for this.

We could *also* define our own exit codes, either test-program-specific or a set of codes for all to use; sysexits.h 
defines EX__BASE as the "base" of the error codes it defines, so we could define a set of error codes from 1 to 63 for 
use by test programs.
_______________________________________________
tcpdump-workers mailing list -- tcpdump-workers () lists tcpdump org
To unsubscribe send an email to tcpdump-workers-leave () lists tcpdump org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s