RPC analysis

[Charles Hannum <abuse () spamalicious com>]

The scope of this bug is somewhat limited.  It depends on a particular
multiplication overflowing:

        nodesize = c * elsize;

In order for this to happen, you must have
maxsize>max(nodesize)*elsize; otherwise the c>maxsize check will catch
it.  Hence the patch:

-       if ((c > maxsize) && (xdrs->x_op != XDR_FREE)) {
+       if ((c > maxsize || UINT_MAX/elsize < c) &&
+           (xdrs->x_op != XDR_FREE)) {

In practice, there are only two ways that this constraint is violated:

1) An application explicitly passes in a large maxsize.  I generally
   consider this an application bug, in the same vain as sending bogus
   arguments to calloc(), fread()/fwrite() or qsort().  However, amd
   and various amd utilities -- amd, amq, pawd, and probably others --
   pass in a maxsize ~0 (i.e. all-1s) and are therefore susceptible.

2) A program uses rpcgen with unbounded arrays; e.g. in rex.x:

        rexstring rst_cmd<>;    /* list of command and args */
        rexstring rst_env<>;    /* list of environment */

   (Note that nothing in our source tree actually *uses* rex.x, but
   the same problems occurs in rusers.x.)

There are only a few things in our source tree that actually use
xdr_array():

/usr/src/dist/am-utils/amd/amq_subr.c
/usr/src/dist/am-utils/amq/amq_xdr.c
/usr/src/dist/am-utils/libamu/xdr_func.c
/usr/src/gnu/dist/toolchain/gdb/vx-share/xdr_ld.c
/usr/src/gnu/dist/toolchain/gdb/vx-share/xdr_rdb.c
/usr/src/lib/libc/rpc/authunix_prot.c
/usr/src/lib/librpcsvc/rnusers.x 
/usr/obj.i386/lib/librpcsvc/rex.c
/usr/obj.i386/lib/librpcsvc/rnusers.c
/usr/obj.i386/lib/librpcsvc/rusers.c 
/usr/obj.i386/usr.sbin/rpc.pcnfsd/pcnfsd_xdr.c

The GDB stuff and rex.c are not actually linked into anything.  The
uses in authunix_prot.c, rnusers.c and pcnfsd_xdr.c are all
sufficiently bounded.  Ergo, the only vulnerable programs are
amd/amq/etc. and rusers (the client, not the server).