nmap-2.54b5+V-CVS (I know... I know... BETA6...)

["Jay Freeman \(saurik\)" <saurik () saurik com>]

Oh, god… :).   OK, I have a lot of notes to say about _this_ one.  First
off, sorry it took so long to get _something_ supporting nmap-2.54b5 (which
was a semi-important bug fix, wasn’t it?).  The main reason is actually
because I kept putting off putting some form of licenses on the new
nmap-related-yet-not-really code sections.  The release of nmap-2.54b6 got
me off my ass about that… semi-interesting note:  when Fyodor released
nmap-2.54b6 I almost started beating my head on my desk (more information
available below).



About a month ago I decided I would open up my CVS servers once I got more
time to do active development on nmap, so here it is.  I don't have a new
release for anyone right now, things are a little too busted.  I started
making an omelettes, and realized I would have to break a few eggs, but
accidently dropped them all on the floor and need to wait until I can buy a
few more eggs....  Besides, I didn't make much headway to the stuff most
people are interested in.  End result?  Here it is, and it isn't all that
pretty (more information available below).



All right, what we have here:



Not much changed on the version scanning front.  I am going to go out on a
limb here and say I changed nothing, I really don't remember, but it
definitly wasn't an interesting change if there was one.  This release is
all about moving towards modularizing existing functionality, including the
version scanning into a new application/library called "anakrino" (which
doesn't work yet, totally defeating the purpose, so nmap+V is still there
with its built-in nmap modifications).




This is the biggee:  table based output.  I got mad, and wrote a library I
don’t even like anymore called "libtabular" that let’s me multiplex my
output to all sorts of different file formats.  Because of this, nmap+V now
has HTML and XML output.  Yes, that’s right:



__ Fyodor and I both independently added XML output. __



ACK!  I didn't think he would ever get around to it :).  Mine’s more general
(single API, every file format, hardly any "if its this file format, do
this, but otherwise do that", few places it is like that are to support
places where existing output methods showed different data, like the exact
time of a scan), but Fyodor’s is much "better" (concentrated more time on
the output itself than the process of outputting it).  My goal was to get
any output at all (as libtabular didn't even work yet), and then modify
_all_ the output forms at once until the XML output looked better.  I even
managed to get libtabular supporting “grepable output” (as Fyodor now calls
it... I don't really like the name as I never really felt like I could
safely grep that for most purposes... maybe a few sed’s?  one to get it onto
multiple lines, another to break out the ‘/''s?... the XML is almost better
for that, run it through xml2 and start scripting away)....  I wanted to
send an e-mail to Paul (Tod Rieger) about this stuff, who had sent me an
e-mail asking about something and making a few general suggestions, but his
e-mail address didn’t work….



Through libtabular, I started modularizing stuff.  A new directory that
separates out the scans, and source files for them.  Not quite what I want
currently, going to keep playing with it.  The files have output commands
that libtabular calls as it needs more output to fill in outstanding tables.
The outputted columns can then be registered dynamically.  Currently they
are only being registered by the function that used to do the per-host
output (I renamed it, and don’t remember what it used to be called).  The
goal would be to get this to a stage where Fyodor likes the organization (I
don’t even yet…) and then get him to move pos_scan and super_scan out into
the separate source files on his end so I can start work on _really_
modularizing stuff (I don’t like making changes to nmap that are _too_
extreme in my CVS for fear of later updates from Fyodor driving me… well…
insane because I can't easily add his new changes).



I actually thought it was kind of funny that I added a few new libraries for
the same release that Fyodor did (had this stuff ready before nmap-2.54b5),
and ended up making quite the same changes to add the new library to the
build scripts….  CVS actually thought we were going in similar directions
and that we just thought of different names for our stuff :).



Oh, and I moved the regex stuff into nbase, where it seemed more
appropriate.



BTW, I have _no_, I repeat _no_ idea where _this_ stuff builds.  I’m not
even going to mention Solaris or FreeBSD this time (DOH, already did… well…
ignore that…).  Just remember, it _is_ quasi-released CVS code, after all
:)... I haven't even tested to make sure everything compiles OK with g++ yet
(kept forgetting, will try to do that more often from now on).  BTW, the way
I currently think of this is "super experimental branch of nmap to see what
kind of neat features can be added without screwing up the general
population".  The flag of nmap+V doesn't really fit this anymore (and never
really fit my CVS server, which is also where I originally built the nmap
g++ patches).  All the versions of nmap+V are tagged and accessable, as well
as



CVS Info:



Server: saurik.com

User: cvs

Password: cvs

Repository: /cvs/nmap

Module: nmap



Obligatory CVS command reference:



Get the source code:

cvs -d :pserver:cvs () saurik com:/cvs/nmap login

<password is "cvs">

cvs -z3 -d :pserver:cvs () saurik com:/cvs/nmap co nmap



Later, from the new directory, to get new files:

cvs update -dP



Check for all the changes you have made to the code:

cvs diff -u



Get a list of log entries for a file:

cvs log <filename>



I will probably move these to another box off of anakrino.org in the next
week or so... I kind of firewalled myself out of that one yesterday... will
need to wait until Monday when someone is in the office :).  That's the one
with the nice, new version of CVS and the really secure environment for it,
etc.  For right now I just made myself a few backup copies of the code....
CVS currently doesn't have the protocol database for anakrino (not that
anakrino can read it, or even that much is in the protocol database, so it
doesn't matter).



I hadn't commited my really messed up nmap.c in a while... but... well, its
there if anyone wants to look at what its outputting and make comments (like
"roll this change back, NOW!!!! and port to nmap-2.54b6 IMMEDIATLY!").
Seriously, I like comments... comments are good.  Moving to nmap-2.54b6 is
going to be a relatively slow process as it currently stands, as I removed
almost all of the old output code and rebuilt it all... I'm probably going
to work on the next output idea of mine, and then mess with it.  Maybe I
should just back port the Red Hat 7 change, as that's the only thing that
even applies anymore....  BTW, what happens when it gets to the point where
Fyodor makes a release that doesn't even affect any of my code?  Oy ve...
that's going to be funky.  Here is what I finally decided I wanted to do (as
of a week or so ago):



_Only_ have XML output from the internals of nmap.  Not even bother
outputing other formats.  Instead of having a generalized structure for
building all the output formats at once (libtabular), build a generalized
system for generating a certain version of a certain type of output through
chains of XSLT.  This is a more generalized way of doing what I suggested
before in the XML output discussions for allowing drastic changes to the XML
format (regular changes, adding this here, adding that there, etc. are
handled by the shear fact that it is XML, after all).  So if you wanted
grapable output format version 1.1, I could see that "oh, we have an XSLT
here from the XML format version 1.2... currently we are on XML format
version 1.3, but that's all right, we will run the XML-1.3->XML-1.2 step and
then do XML-1.2->Grep-1.1".  Not sure if libxml is going to be able to do
that, however.  Might end up being forced to finally fork and say "C++ past
here".  Although that leads to the question of "why am I building this"?
The main reason for spending so much time on the output methods is the hope
that it could improve nmap, not allow me to splinter off a bunch of versions
of different output formats that are never going to get used in applications
anyway (hehe).  In fact, it gets worse, as the biggest problem to inclusion
isn't even the C++ but the Xerces (Apache's really powerful XML library)
dependancy :(.  The web server for www.libxml.org doesn't seem to be working
well today, so I will have to put off checking for in memory multi-level
XSLT support later.



I really think this method would work better.  That way the XML output could
be designed to be as sophisticated as is neccessary, and we just need to get
XSLT to the other formats (which shouldn't be difficult, I write XSLT all
the time, and being the person who ends up writing the functionality, and
probably the only person to ever use it, I guess I would be in charge of
that).




Sincerely,

Jay Freeman (saurik)

saurik () saurik com