Re: [RFC][patch] XML structured script output (summary of output changes)

[David Fifield <david () bamsoftware com>]

On Mon, May 21, 2012 at 08:57:38PM -0500, Daniel Miller wrote:
> Congratulations everyone on version 6!
>
> For a year or so, I've been looking for feedback on my patch for XML
> structured output for NSE scripts [1], and as it has been a while, I
> thought I'd bring it up again. I've fixed a couple bugs since last
> time, and I've started keeping my local copy of the Nmap source under
> Git source control, so I can rebase my branch as development continues
> on the official source. I'm attaching 2 patches: nse-structured.diff
> has all the code changes to use XML output for scripts, and
> scripts-structured.diff patches a few scripts to take better advantage
> of the structured output.
>
> As far as impact to script writers, there shouldn't be much, since a
> table passed to stdnse.format_output() will work as it always has, as
> will simply returning a string from a script. In this regard, the
> functionality is largely opt-in. If the patch is accepted, I'll spend
> time updating existing scripts to structure their output.
>
> There is room for improvement: Currently, "horizontal lists" don't
> work; that is, comma-separated values on one line of normal output
> should be output as a list of <elem> elements in XML. The base
> functionality is there, with ScriptOutputNode.set_display(FLOW), but
> the logic to tie it in as a Lua function is missing. I don't figure
> this is a big deal, and can be added in later.

Thank you for following this so persistently, Dan. I think we are close
to finally getting something merged. There is a lot to discuss so I'm
going to write several messages. This one is a summary of the output
changes made by your patch.

As I understand the patch, it co-opts stdnse.format_output and uses it
as the mechanism to output structured XML. The XML structure will mirror
that of the table given to stdnse.format_output. However, scripts that
use format_output merely as a way to output some nested lines, need to
be modified to break their output into more structured name-value pairs,
otherwise the structured XML output is essentially the same sequence of
lines.

Here is normal screen output with some typical script results.

# nmap -6 -n -Pn -p443 --script=address-info,ssl-cert fe80::1e6f:65ff:feb6:cad3 ::1
Nmap scan report for fe80::1e6f:65ff:feb6:cad3
Host script results:
| address-info: IPv6 EUI-64
|_  MAC address: 1c:6f:65:b6:ca:d3 (Giga-byte Technology Co.)
Nmap scan report for ::1
PORT    STATE SERVICE
443/tcp open  https
| ssl-cert:
|   Subject: commonName=localhost
|   Issuer: commonName=localhost
|   Public Key type: rsa
|   Public Key bits: 1024
|   Not valid before: 2012-06-14 01:05:29
|   Not valid after: 2013-06-14 01:05:29
|   MD5: 62b8 3671 9c3c fb52 e3e9 4db3 254d d50c
|_  SHA-1: 7514 8e75 f165 d8b5 7cbc 72ae f534 22e1 eb8d d0f4

This is what the XML output looks like now. Script output is stuffed as
a string in //script@output elements.

    <hostscript>
      <script id="address-info" output="IPv6 EUI-64; MAC address: 1c:6f:65:b6:ca:d3 (Giga-byte Technology Co.)"/>
    </hostscript>
    <ports>
      <port protocol="tcp" portid="443">
        <script id="ssl-cert" output="Subject: commonName=localhost&#xa;Issuer: commonName=localhost&#xa;Public Key 
type: rsa&#xa;Public Key bits: 1024&#xa;Not valid before: 2012-06-14 01:05:29&#xa;Not valid after:  2013-06-14 
01:05:29&#xa;MD5:   62b8 3671 9c3c fb52 e3e9 4db3 254d d50c&#xa;SHA-1: 7514 8e75 f165 d8b5 7cbc 72ae f534 22e1 eb8d 
d0f4"/>
      </port>
    </ports>

Here is the XML output using your nse-structured3 patch, without any
script changes. The only change is that output moves from the
//script@output attribute to //script/elem elements.

    <hostscript>
      <script id="address-info">
        <elem>IPv6 EUI-64; MAC address: 1c:6f:65:b6:ca:d3 (Giga-byte Technology Co.)</elem>
      </script>
    </hostscript>
    ...
    <ports>
      <port protocol="tcp" portid="443">
        <script id="ssl-cert">
          <elem>Subject: commonName=localhost
Issuer: commonName=localhost
Public Key type: rsa
Public Key bits: 1024
Not valid before: 2012-06-14 01:05:29
Not valid after: 2013-06-14 01:05:29
MD5: 62b8 3671 9c3c fb52 e3e9 4db3 254d d50c
SHA-1: 7514 8e75 f165 d8b5 7cbc 72ae f534 22e1 eb8d d0f4</elem>
        </script>
      </port>
    </ports>

Here is the XML script output after making format_output changes to
certain scripts. The big change here is that "Label: value" data now
appear as <elem key="Label">value</elem>. Also, multiple lines from
format_output get their own //script/elem elements.

    <hostscript>
      <script id="address-info">
        <container name="IPv6 EUI-64">
          <elem key="MAC address">1c:6f:65:b6:ca:d3 (Giga-byte Technology Co.)</elem>
        </container>
      </script>
    </hostscript>
    ...
    <ports>
      <port protocol="tcp" portid="443">
        <script id="ssl-cert">
          <elem key="Subject">commonName=localhost</elem>
          <elem key="Issuer">commonName=localhost</elem>
          <elem key="Public Key type">rsa</elem>
          <elem key="Public Key bits">1024</elem>
          <elem key="Not valid before">2012-06-14 01:05:29</elem>
          <elem key="Not valid after">2013-06-14 01:05:29</elem>
          <elem key="MD5">62b8 3671 9c3c fb52 e3e9 4db3 254d d50c</elem>
          <elem key="SHA-1">7514 8e75 f165 d8b5 7cbc 72ae f534 22e1 eb8d d0f4</elem>
        </script>
      </port>

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/