exit status codes semantics in tests

[Denis Ovsienko <denis () ovsienko info>]

Hello all.

This started as a comment for libpcap pull request 1787, but it seems
to be useful beyond the scope of the pull request, so let me post this
to the list in case it helps anyone to work on new libpcap tests.
--------------------------------------------------------------------

The current use/non-use of EX_DATAERR (65) and EX_OK (0) in filtertest,
translatetest and enumeratetest, as well as of EX_NOINPUT (66) in
filtertest, has to remain intact because that's how TESTrun tells
whether a test passed or failed, e.g. filtertest usage message says:

Exit status codes:
    0: All input has been successfully processed.
   65: libpcap has orderly rejected the filter.
   66: libpcap has orderly rejected the savefile.
   70: Unexpected libpcap error.
   71: Unexpected OS error.
   74: Unexpected I/O or file format error.
   64: This executable has been invoked incorrectly.

Here the currently implemented test logic hinges on the first three
codes and the last four codes just make test failure debugging easier.
Other test programs or exit status codes have no impact on the existing
tests.

The semantic difference between EX_USAGE, EX_SOFTWARE and EX_DATAERR
originates in filtertest as the first automated test program and
generally implements the following logic (the description may be not
entirely accurate, but should convey the idea), which allows to tell
expected failures of reject tests from unexpected failures that account
to transient host effects and bugs.

EX_USAGE concerns only the composition of command-line arguments (which
leads me to notice that parsing of the DLT does not conform to this
rule), so if the use case demands a filter expression, but it is not
given as an /opaque/ argument, filtertest exits with EX_USAGE to
indicate that it is impossible to produce the test result.  The root
cause is in TESTrun.

Alternatively, it can exit with, e.g. EX_IOERR to indicate that the
specified filter expression file could not be read, thus it is
impossible to produce the test result.  The root cause is most likely in
the filesystem.

Otherwise, if pcap_open_dead() fails, filtertest exits with EX_SOFTWARE
to indicate that it is impossible to produce the test result.  The root
cause is in libpcap.

Otherwise, it can exit with EX_DATAERR to indicate that the opaque
piece of data is present and libpcap works and the result is that
libpcap rejects the data.  The root cause is in the filter expression
and any other test parameters.  Whether this is the expected result for
the test is up to the test type and definition in TESTrun.

Optionally, it can exit with EX_NOINPUT to indicate that libpcap has
failed opening the specified savefile.  The root cause could be a
filesystem error or a malformed file.

Otherwise it prints the filter program or the filter program's return
value for every packet in the savefile, and exits with EX_OK.  The only
other code path that results in EX_OK is "filtertest -h".

Eventually translatetest and enumeratetest followed this model to the
extent of their use case.  E.g. it is a job of
pcapint_get_decuint_noendp() to reject a malformed argument, it is a
job of test_pcapint_get_decuint_noendp() to turn that into EX_DATAERR,
and it is a job of translatest.c to exit with EX_USAGE unless some
/opaque/ argument has been given to test_pcapint_get_decuint_noendp().

-- 
    Denis Ovsienko
_______________________________________________
tcpdump-workers mailing list -- tcpdump-workers () lists tcpdump org
To unsubscribe send an email to tcpdump-workers-leave () lists tcpdump org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s