mailing list archives
Re: Suggestion: Using script output - Human-readable and machine-parseable
From: Daniel Miller <bonsaiviking () gmail com>
Date: Thu, 7 Apr 2011 06:54:48 -0500
Thanks for the feedback!
On Wed, Apr 6, 2011 at 5:23 PM, Fyodor <fyodor () insecure org> wrote:
For comparison's sake, another option we could consider is formatting
script results in XML (for structured XML output) and a separate
human-readable format (as we do now) for normal output.
One advantage of having scripts emit YAML is that it is more
structured than currently, which means it can be converted to other
formats (such as XML) with minimal opaque blobs.
The big advantage of YAML is that it is one format which can serve
both purposes. This would make it easier for script writers to use,
and would also allow folks to more easily parse the script output from
Nmap's normal-format output.
But my worry is that this single format might be worse for both
purposes. The XML output (which is the version meant for parsing)
will be harder to parse because programs doing so will now need a YAML
parser in addition to an XML parser. We would have to add that to
Zenmap and Ndiff if we want to make use of the results, and so would
3rd party programs which use Nmap.
As I mentioned to David earlier, there is an accepted mapping from
YAML to XML that we could do internally. I think this is the next
thing I'll tackle: coercing the XML output routines to emit XML based
on the script's output.
Meanwhile, the user-visible output is likely to suffer to because it
would be constrained to the YAML format. So, for example, we end up
with ugly labels for a bunch of elements even when it is obvious to
the user what the elements represent.
YAML is not as constraining a format as you might think. I realize
most of the examples I showed use "ugly labels," but that's just for
mapping types (hash table/hashmap/associative array). Sequence types
(list/array/set) are also supported. Basically, if it can be stored in
a Lua table, it can be displayed in YAML.
For these reasons, I wonder if it would be better to create a general
results formatting function which takes a Lua table of results and
creates both XML and normal output from that?
As I read your feedback, one of the more drastic changes I considered
was changing the interface between the scripts and the Nmap program.
Right now, scripts return a string, which Nmap has to make work in
each output format. For this case, YAML makes sense, since it is both
a string and a parseable representation of a data structure. Instead,
we could have scripts return a table, which Nmap would then format
differently for different output formats. The trouble is, no matter
what, there has to be a linkage in the middle. Either Nmap has to
parse YAML from the scripts, or it has to know how to handle a Lua
table for output.
All of Nmap's output could potentially be formatted in YAML, but our
approach has instead been to make the normal output as human-readable
as possible and to provide parallel results in XML for parsing.
I certainly understand that we wouldn't want to convert all output to YAML.
I haven't thought about this issue as much as others have, and my YAML
knowledge is basically limited to Daniel's previous posts on the topic
and the example script output in his emails. So I'm not against the
YAML approach if it proves to be the best. I just want to see what
ideas people have about the potential problem of YAML being harder to
read in human-readable output and harder to parse when embedded in
XML. I'd also like to hear what people think about the option of
writing a function/library which takes a Lua table of results data and
outputs it in nice user-readable form (like we already have) and also
I'll try to get the XML translation part done in the next week or so,
and we'll see if you have the same concerns. One issue I anticipate is
that the less-structured nature of script output means we can't write
a DTD that defines Nmap's XML output anymore. I don't know as much
about that, but if someone could offer input on how to define a schema
that includes arbitrary tags, that'd be great.
Sent through the nmap-dev mailing list
Archived at http://seclists.org/nmap-dev/