Re: exit status codes semantics in tests

[Denis Ovsienko <denis () ovsienko info>]

On Fri, 16 Oct 2026 14:41:07 -0700
Guy Harris <gharris () sonic net> wrote:

> On Oct 16, 2026, at 2:05 PM, Denis Ovsienko <denis () ovsienko info>
> wrote:
>
> > On Fri, 16 Oct 2026 10:36:49 -0700
> > Guy Harris <gharris () sonic net> wrote:  
>
>       ...
>
> > If TESTrun has received EX_USAGE from a test program, it means a
> > bug in TESTrun because every command line is an internal product of
> > TESTrun.  
>
> I.e, EX_USAGE should not be used for any cases where a command-line
> argument is *intentionallY* made incorrect in TESTrun because the
> program is being used to test to make sure that a particular argument
> fails - whether it's an invalid filter expression for the particular
> link-layer type (or for *any* link-layer type), or to make sure that
> an invalid string name for a link-layer type returns -1 from
> pcap_datalink_name_to_val(), or....

DLT name-to-number translation requires more attention.  I do not
remember if the choice of EX_USAGE instead of EX_DATAERR achieved some
specific useful effect or just stayed this way since early revisions.
This translation is not the main focus of this test program, since
recently there is another test program with separate tests that cover
this corner of the API specifically.  filtertest.c may not require a
different status code for DLT, but it certainly requires a comment
there.

In filter testing, however, "correct" has layers of interpretation.
One is that EX_USAGE concerns only presence of a filter in getopt()
sense, not its meaning.  Another is that the filter can compile
"successfully" when it is not supposed to, which would mean success for
filtertest, but a test failure in TESTrun.  Or it could indeed compile
successfully as expected, but the resulting filter program would be not
the expected one (a compiler bug).  Or it could be the expected program,
but the attempt to filter packets using that program would not produce
the expected results (an interpreter bug).

To rephrase this, a test program returns one of "data success", "data
failure" and "some other error", but the only direct correlation is
"other error".  Only TESTrun has enough information to tell whether the
test has passed or failed based on "data success", "data error", any
non-exit-status-code output, the expected result(s) and the type of
test being run.

> > Time stamps in filter apply tests have been discussed once or twice
> > before, this could use a command-line option.  Other packet
> > meta-data could use the same or other options.  To make complete
> > sense, this would need meaningful savefiles and meaningful test
> > definitions to exercise specific code paths.  
>
> filtertest is a program for testing filters, not to test capture-file
> reading code; giving it the responsibility for the latter would make
> it even *more* convoluted. I'm working on a *separate* program for
> that.
>
> Unless there are filter expressions that check the packet arrival
> time, there's no need for the filter tests to care about it.

Let's see how it works.

> > For packet data it is not clear what code path it would test.  
>
> The code paths that reads pcap and that read pcapng files and produce
> a struct pcap_pkthdr and a lump of packet data.
> > Yes, filter expressions can and in the TESTrun case do come from a
> > temporary file.  What matters is that sometimes the result of a test
> > must be conveyed as "could not run the test at all", not "test
> > passed" or "test failed".  
>
> With EIO, ETIMEDOUT, and possibly other read errors being "could not
> run the test at all".


-- 
    Denis Ovsienko
_______________________________________________
tcpdump-workers mailing list -- tcpdump-workers () lists tcpdump org
To unsubscribe send an email to tcpdump-workers-leave () lists tcpdump org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s