Re: Suggestion: Using script output - Human-readable and machine-parseable

[Daniel Miller <bonsaiviking () gmail com>]

Thanks for the feedback!

On Wed, Apr 6, 2011 at 5:23 PM, Fyodor <fyodor () insecure org> wrote:

> For comparison's sake, another option we could consider is formatting
> script results in XML (for structured XML output) and a separate
> human-readable format (as we do now) for normal output.

One advantage of having scripts emit YAML is that it is more
structured than currently, which means it can be converted to other
formats (such as XML) with minimal opaque blobs.

> The big advantage of YAML is that it is one format which can serve
> both purposes.  This would make it easier for script writers to use,
> and would also allow folks to more easily parse the script output from
> Nmap's normal-format output.
>
> But my worry is that this single format might be worse for both
> purposes.  The XML output (which is the version meant for parsing)
> will be harder to parse because programs doing so will now need a YAML
> parser in addition to an XML parser.  We would have to add that to
> Zenmap and Ndiff if we want to make use of the results, and so would
> 3rd party programs which use Nmap.
As I mentioned to David earlier, there is an accepted mapping from
YAML to XML that we could do internally. I think this is the next
thing I'll tackle: coercing the XML output routines to emit XML based
on the script's output.

> Meanwhile, the user-visible output is likely to suffer to because it
> would be constrained to the YAML format.  So, for example, we end up
> with ugly labels for a bunch of elements even when it is obvious to
> the user what the elements represent.
YAML is not as constraining a format as you might think. I realize
most of the examples I showed use "ugly labels," but that's just for
mapping types (hash table/hashmap/associative array). Sequence types
(list/array/set) are also supported. Basically, if it can be stored in
a Lua table, it can be displayed in YAML.
> For these reasons, I wonder if it would be better to create a general
> results formatting function which takes a Lua table of results and
> creates both XML and normal output from that?
As I read your feedback, one of the more drastic changes I considered
was changing the interface between the scripts and the Nmap program.
Right now, scripts return a string, which Nmap has to make work in
each output format. For this case, YAML makes sense, since it is both
a string and a parseable representation of a data structure. Instead,
we could have scripts return a table, which Nmap would then format
differently for different output formats. The trouble is, no matter
what, there has to be a linkage in the middle. Either Nmap has to
parse YAML from the scripts, or it has to know how to handle  a Lua
table for output.

> All of Nmap's output could potentially be formatted in YAML, but our
> approach has instead been to make the normal output as human-readable
> as possible and to provide parallel results in XML for parsing.
I certainly understand that we wouldn't want to convert all output to YAML.

> I haven't thought about this issue as much as others have, and my YAML
> knowledge is basically limited to Daniel's previous posts on the topic
> and the example script output in his emails.  So I'm not against the
> YAML approach if it proves to be the best.  I just want to see what
> ideas people have about the potential problem of YAML being harder to
> read in human-readable output and harder to parse when embedded in
> XML.  I'd also like to hear what people think about the option of
> writing a function/library which takes a Lua table of results data and
> outputs it in nice user-readable form (like we already have) and also
> XML.
>
> Cheers,
> Fyodor

I'll try to get the XML translation part done in the next week or so,
and we'll see if you have the same concerns. One issue I anticipate is
that the less-structured nature of script output means we can't write
a DTD that defines Nmap's XML output anymore. I don't know as much
about that, but if someone could offer input on how to define a schema
that includes arbitrary tags, that'd be great.

Dan
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/