Re: NSE structured output--normal output formatting

[Daniel Miller <bonsaiviking () gmail com>]

On Fri, Jul 20, 2012 at 11:43 AM, Patrick Donnelly <batrick () batbytes com> wrote:
> On Fri, Jul 20, 2012 at 10:56 AM, David Fifield <david () bamsoftware com> wrote:
> > This doesn't make sense to me. If the script can provide a function that
> > converts its output to a string, it can also call that very same
> > function to produce a string. (That's all NSE is going to do, anyway.)
> >
> > More importantly, the two models do not only differ in who gets to call
> > the function. A formatting callback or metamethod is strictly less
> > flexible. It would require rewrites of many existing scripts that build
> > up their string output incrementally while running.
> >
> > Further, I see a formatting callback as requiring hundreds of lines of
> > new code, not dozens, and being harder to explain to script developers,
> > especially if hidden in a metamethod.
>
> I think what is key to deciding is whether or not it should be
> possible to recover normal output from the XML output. My belief is
> that, yes, you should be able to. With this in mind, I think it's
> important that scripts add a serialization function (which I think
> will be the exception rather than the norm!) to some part of their
> structured output table. With this function (which may be __tostring
> or some other method), we can encode a whole sub-tree as "normal"
> output which can be attached to the element. For example, in the
> nfs-ls script:
>
> Normal output:
>
>         drwxr-xr-x  1000  100   4096     2010-06-17 12:28 /mnt/nfs/files
>
> XML output:
>
> <table normal="drwxr-xr-x  1000  100   4096     2010-06-17 12:28
> /mnt/nfs/files">
>     <elem key="perm">1755</elem><elem key="uid">1000</elem>...
> </table>
>
> Here the normal tag will allow recovery of normal output and prune
> that entire sub-tree from consideration (all of the elem tags).
>
> This won't be possible if we encourage scripts to have two different
> encodings for output (string and structured). A machine reading the
> XML output can't infer how to print the file as one line and having
> the entire output in the string@output tag doesn't help either.
>
> [Another option is to add a Lua function to the XML which does the
> conversion. That's a bit extreme though.]
>
> --
> - Patrick Donnelly
> _______________________________________________
> Sent through the nmap-dev mailing list
> http://cgi.insecure.org/mailman/listinfo/nmap-dev
> Archived at http://seclists.org/nmap-dev/

I don't think that David is advocating for string output that is
wholly (or even mostly) different in content from the structured
output, but rather that the transform function should be called within
the script body, rather than by the engine, after the fact. For that
matter, being able to build the string as each component is
collected/extracted, rather than in a separate function at all.

I am not advocating for abandoning a string representation that cannot
be recreated from XML structure: there are too many display options
for it to be obvious how to generate the string. That is why the
@output attribute of the script tag is still necessary (for recreating
the Normal output from the XML).

I'll throw in one more advantage for the metamethod approach: nesting
of different display types. Consider nfs-ls.nse, which has ls-style
tables of data nested within an outline structure describing each
export. With metamethods, I "tag" each ls-table with the
rows-and-columns-style, then insert each into the standard-style
output table. My implementation on Github can already handle this,
since the recursive algorithm for standard outline-style tables checks
for a __tostring metamethod on each subtable before recursing. This
approach to nesting means that a library function (like mssql) could
even return tables with built-in serialization, making it simpler for
a script author to display results. The flexibility is still there to
extract the data from the library results and reformat some other way.

One way or another, script authors will adapt, so long as the API is
well-designed and documented.

Dan
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/