Home page logo
/

nmap-dev logo Nmap Development mailing list archives

Re: Suggestion: Using script output - Human-readable and machine-parseable
From: Daniel Miller <bonsaiviking () gmail com>
Date: Tue, 5 Apr 2011 07:35:29 -0500

David,

I'm attaching a patch that allows for formatting output as YAML. It
includes a new NSE library, yaml, that is a slightly modified copy of
the libyaml binding from yaml.luaforge.net (properly attributed). The
second diff (luayaml.diff) shows the modifications I made to it.

I added a new function to stdnse, format_yaml, that takes arguments
like format_output, but the tables should be in a different
configuration. I modified smb-os-discovery to show the new way, and
smb-enum-shares to show a half-assed workaround (since a multiline
string can be represented as YAML, it just shoves the existing output
into a YAML block scalar literal). The new yaml-test.nse script is
just to demonstrate and test the output, since it's a prerule script
and can be run without running a scan.

As it currently stands, I'm dynamically linking against libyaml, and
using systemwide headers for it, which I know is not the way we'll
want to go, but it was easiest to start. I haven't worked with
autoconfig before, so I didn't know how to package the library with
Nmap. It's pretty stable (0.1.3 was released August 2009), and not too
big, so it wouldn't be tough to include it. If you want to test, the
libyaml-0-2 and libyaml-dev packages for Ubuntu were sufficient to
build and run my patched version.

For tagging the output as YAML, a YAML document always starts with
"---", so that turns out to be a good marker to look for. The way Nmap
prefixes script output with "| " makes it impossible to copy-and-paste
the output as valid YAML, though. Not a huge deal, since XML output
doesn't do that, though. Since YAML is structured already, it wouldn't
be necessary to convert it to an XML element hierarchy, but if we
wanted to (to save writers of Nmap XML parsers from needing a YAML
library), see http://yaml.org/xml/index.html for the generally
accepted way of doing it.

Dan

On Sat, Apr 2, 2011 at 11:29 AM, David Fifield <david () bamsoftware com> wrote:
On Thu, Mar 31, 2011 at 03:46:41PM -0500, Daniel Miller wrote:
Hey list,

From the Nmap TODO:
o Nmap should have a better way to handle XML script output.
 o We currently just stick the current script output text into an XML tag.
May I make a suggestion? If we format script output as YAML
(http://www.yaml.org/), it will be machine-readable, while preserving
the human-readable nature. There are a couple of YAML bindings for Lua:
libyaml (http://yaml.luaforge.net/) and libsyck
(https://github.com/indeyets/syck/tree/master/ext/lua)

The quickest way to support this straight off would be to modify
stdnse.format_output to output YAML. The NSE runtime could quote
non-YAML return values as block-literal scalars, until such time as they
are rewritten to return more useful formatted output. A downside would
be that output would take a few more lines than before, though this
could be overcome by using "flow styles," of which JSON is a proper
subset.

Benefits of doing this are:
* extensibility without requiring modifications to Nmap XML schema
* Human-readable nature of YAML means a single output format for human
and machine consumers
* existing structured-output using stdnse.format_output means very few
changes for script authors
* YAML bindings available for Lua, Python, Perl, Ruby, C, and many other
languages.

I was going to submit this idea with a patch, but I realized I don't
have the first clue as to how to integrate a new Lua library binding
into the Nmap source. Instead, here's some examples of YAML-formatted
script output.

How about this: Write a new stdnse.yaml function with an interface
similar to that of stdnse.format_output. Then we can switch scripts one
at a time instead of all at once.

What I am thinking is, that stdnse.yaml can somehow tag the output as
being structured data and not just plain text. Then we can modify the
XML output to look for this tag and emit an element hierarchy instead of
(or in addition to) a chunk of text. But that's a second step; the first
is just getting YAML on the screen.

Take a look at json.generate, which does something similar.

David Fifield

Attachment: nse-yaml.diff
Description:

Attachment: luayaml.diff
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]
AlienVault