Home page logo

nmap-dev logo Nmap Development mailing list archives

Re: [RFC] NSE pack/unpack library
From: "Philip Pickering" <pgpickering () gmail com>
Date: Fri, 20 Jun 2008 16:24:56 +0200


2008/6/20 Fyodor <fyodor () insecure org>:
 I assume that the lpack library
you are starting with is as described at:


yes, the second one (the first is an outdated derivate, IIRC)

Are there reasons that we can't use the current lpack library as-is?
When we incorporate third party libraries, it is best to reduce
changes to the minimum necessary.  Among other advantages, that makes
it easy to upgrade them.  When we fix or improve things, it is best to
submit those to the upstream maintainers in the hope that they will
incorporate our changes so we don't have to keep maintaining them.

Of course, sometimes we need functional changes which aren't desirable
upstream.  And sometimes upstream is not maintained and so we end up
maintaining all our changes.  But we try to limit such cases.

Also, I'm not sure there is any need for Perl format compatability,
unless the Perl version is much more powerful in a way which is
important to us.  It may seem that we prefer Perl, since we have a
Perl Compatable Regular Expression library in NSE.  But that was done
because PCRE was already part of Nmap when NSE was added, and Lua's
"patterns" just didn't seem powerful enough for our needs.

Another advantages of limiting the ways that our version diverges is
that we may be able to simply reuse or link to the source library's

Now I don't know anything about lpack and I rarely use Perl's
pack/unpack.  So maybe there are important reasons why we need to use
Perl-style.  But if there aren't, I think we should stick with the
format of the source library we use.

Well, I understand your concerns. But I try to explain, why modifying lpack
seems necessary to me.

lpack lacks some functionality we need. These features are
at least H for hex strings, B for bit strings and x for null/ignoring bytes.
By adding these features the original operator characters get misleading.

For example, the original short-operator was 'h'/'H' instead of 's'/'S'.
If I add a hex operator and call it 'H', we have a problem. If I call it
'X' and the null-byte 'n' to avoid confusion, then we also have a problem,
since 'n' is also already used by lpack as a lua native number.
Of course it's possible to call heX 'X', nUll byte 'U' and biT strings 'T', it
just doesn't seem intuitive anymore (but that's just personal taste).

Still, the main problem is the missing hex, bits and the null byte, rather
simple features I believe, but still needed.

Therefore adding lpack without modifications doesn't seem to be enough
(the same is true with an alternative pack/unpack lib called struct).
Especially since lpack registers itself at string.pack and string.unpack, which
folks at #nmap-nse opposed, at least minor changes are necessary.

Another interesting detail is the possible addition of pack/unpack
to Lua 5.2, using either lpack or struct as a base (again putting it
under string.*, as the discussion on the Lua mailing list suggests)
which will most likely miss the features, too.

I guess, we have several alternatives:
- modify lpack, send it to its maintainer and hope the features get added
- use lpack unmodified (except registering in under bin.*) without new features
- use lpack unmodified, write a wrapper around it and add needed features there

Right now I'm not really enthusiastic about the last alternative. Although
it solves both requirements (adding features and leaving as much
as possible unmodified) it does this by adding quite a bit of overhead.

The first advantages/disadvantages which come to mind are

(+) upgrading to a new lpack or pack/unpack in 5.2 would be rather easy

(-) it results in some kind of complicated design, calling a function to call
another one, both working on different aspects of the same job. But then
again it could be argued, that this is just the way functions work most of
the time :)

I hope the situation got clearer now. I'd like to hear your opinion on it.


Sent through the nmap-dev mailing list
Archived at http://SecLists.Org

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]