Re: Nmap-NSE Release candidate 2

[doug () hcsw org]

Hi Diman!

On Mon, Aug 07, 2006 at 02:53:16PM +0200 or thereabouts, Diman Todorov wrote:

> And remember, I always jump up and down with joy when I receive  
> feedback, regardless
> if positive or negative ;)

I'm glad to hear that except I'm afraid you might be doing a lot of jumping
up and down as this is a very long piece of feedback. :)

I just downloaded your latest NSE distribution. It works well and I look forward
to having it in the main distribution. Here are some of my comments:



I don't know if there are any plans to add user-specifiable parameters
into NSE but I have a feeling that eventually we will run into this sort of need.

As an example, what do we do when somebody writes a cool NSE script that checks
an FTP server for a specific vulnerability but, oh no, the user needs to be able
to specify a user/password combination before the vulnerability can be tested?

Do we make the user edit the lua source file? Do we add some means to pass
parameters through Nmap into the lua script? Or do we simply consider these
cases beyond the scope of NSE?

A conceivable plan could be

nmap -sC --script=whatever.lua --script-params='user="desmond",passwd="molly"' target

which could bind 'user and 'passwd in the lua environments.




I noticed in many of your portrule tests you require the numerical port
of the service AND the service string to match. For example:

chargenTest.lua:portrule = function(host, port)
chargenTest.lua-        if      port.number == 19
chargenTest.lua-                and port.service == "chargen"
chargenTest.lua-                and port.protocol == "udp"
chargenTest.lua-        then

daytimeTest.lua:portrule = function(host, port)
daytimeTest.lua-        if      port.number == 13
daytimeTest.lua-                and port.service == "daytime"
daytimeTest.lua-                and port.protocol == "udp"
daytimeTest.lua-        then

echoTest.lua:portrule = function(host, port)
echoTest.lua-   if      port.number == 7
echoTest.lua-           and port.service == "echo"
echoTest.lua-           and port.protocol == "udp"
echoTest.lua-   then

Since these are some of the simplest NSE scripts we can probably expect them
to be used as examples and skeletons for all the scripts to come. For instance,
Marek's excellent new ftpbounce.lua script:

ftpbounce.lua:portrule = function(host, port)
ftpbounce.lua-  if port.number == 21
ftpbounce.lua-     and port.service == "ftp"
ftpbounce.lua-     and port.protocol == "tcp"
ftpbounce.lua-  then

If I'm reading this code right, these scripts won't be run against
discovered services that version detection finds them on a non-standard
port. Although this probably isn't that big a deal for chargen and its ilk,
I can certainly envision somebody wanting to test FTP servers running
on unpredictable ports.

The other method used in your scripts, Diman, will probably be more effective
because it will be applied depending on the service detected. For instance:

showHTMLTitle.lua:portrule = function(host, port)
showHTMLTitle.lua-      if
showHTMLTitle.lua-              (       port.number == 80
showHTMLTitle.lua-                      or port.service == "http")
showHTMLTitle.lua-              and port.protocol == "tcp"

This should run on any port that version detection discovered to speak HTTP.
Web-servers embedded in an incredibly diverse range of different devices listen on
wacky random ports all the time - and hundreds (thousands?) of them are identified by
version detection to be HTTP. This is a great symbiosis of version detection and NSE
and has huge future potential!

But, by the same token, if port.number is 80 and port.service is known to *not* be http,
do we really want to run this script? I think it might be better to always ignore
port numbers and only depend on the service name + transport protocol.

One of the biggest benefits of version detection is that it makes people realise
just how little security benefit they get from obfuscating port numbers and runnning
services on non-standard ports. In fact, this is where much of the version detection
effort has been focused: distinguishing and identifying different protocols. I think
NSE could take advantage of this information very nicely - especially in the HTTP
examples.

Accounting for just the service name should also add another layer of stability
and assurance to somebody running or writing lua scripts: version detection has
verified that, at least superficially, this port seems to speak the service we're
hoping it to speak.



Perhaps a useful abstraction that could be made is to have a shortcut
in portrule. For instance, what if portrule could be either a function
OR a string? First of all, thank lua's dynamic typing system. Then
consider being able to specify portrules like the following:

portrule = "http/tcp"

instead of the longer

portrule = function(host, port)
  if port.service == "http"
     and port.protocol == "tcp"
  then
end

Naturally some portrules would still require functions for more elaborate
tests.




Until I looked at the lua source code, it wasn't clear to me that port 113
needed to be scanned and determined to be an auth service before the
showOwner.lua script would run. Perhaps some sort of documentation or warning
is appropriate? Also, in a similar vein to above, what if we (god knows why)
find an auth service on a port other than 113? Couldn't somehow the showOwner.lua
script try the scan with any auth services discovered? And could we make this
general so that this sort of behaviour is automatic (by having some sort
of connect() wrapper that accepts service names instead of ports, for example)?



I know lua provides a fairly complete regular expression library but I still
wonder if it would be worthwhile creating some sort of lua binding for
PPCRE. Nmap requires libppcre for version detection so we can always count on
it being linked. (Interestingly, Nmap will soon have 3 distinct, incompatible
regular expression libraries available - POSIX, PPCRE, and lua's)

I would personally rather use ppcre to create regular expressions for the
following reasons:

o I don't want to learn the syntax and idiosyncracies of yet another regular
  expression library! I've already (more-or-less) committed to memory
  POSIX regexs, Perl regexs, and cl-ppcre's s-expression regexs and I would
  rather not try to stuff another in my poor brain!

o Although I suspect the performance difference for simple patterns is
  pretty negligible, it's never bad to have an efficient regex library
  especially when you can make use of back-patterns, non-greedy matching, etc.

o There might be further opportunities to integrate NSE with version detection
  and other applications that use regexs to match network data. Believe me, the
  perl regular expression format is as close to a standard as you can get.




One of the best things about designing a scripting language extension is that we
can hunt for and discover common scripting idioms and make them conveniently
available to the scripter. In lua, which has no macro capability, this means
adding functions to the default environment. I can see you've recognised this
and have added a number of useful functions for people to use. A good example is
the receive_lines() function documented so:

  status, value = socket_object.receive_lines(n)
    Tries  to  receive  at least n lines from an open connection. On success the
    returned value of status is true and the received data is stored in value.
    If the connection attempt has failed, value contains a description  of  the
    error condition stored as a string.

This is handy because a whole heap of protocols use the concept of "lines" to
divide data and functions like this one make it really easy to read and process
data line-by-line.

But why stop there?

Here is an example from showHTMLTitle.lua:

        result = ""
        while true do
                status, s = socket:receive_lines(1)
                if status == false then
                        break
                end

                result = result .. s
        end

This seems to be a common idiom used in NSE scripts (it already appears verbatim
in several default NSE scripts). It says to receive as many lines as possible from
the target and store all results in a variable bound to 'result.

Why not have a function, say, get_contents() (a la Perl's LWP) so this can be
written like so:

        status, result = socket:get_contents()

Furthermore, when we abstract properly we can start to make efficiency
improvements. In version detection, it isn't necessary to read all the data
from a socket in order to match successfuly. In fact, version detection would
likely take much, much longer if this were the case. In version detection,
the regular expressions are compared after every chunk of data received
so as soon as we have enough to match we finish up. This helps for timeout
purposes, large contents (like super long index.html files on httpds or
annoying FTP banners), laggy network connections, and so on.

If we take this to a further level of abstraction, we could make use of
this same tactic in NSE. Imagine a function

get_contents_until_match = function(regex)
  local result

  while not regex_match(regex, result)
    Keep reading from the socket and concatenating the output onto result
    until regex matches. Once it does, return the result and a good status.
    If it doesn't, or we get some other error from the socket, return
    a bad status.
  end
end

so we can do things like so:

  status, result = get_contents_until_match("^HTTP/1.0 \d\d\d .*\r\n")

  if (status == true) then
    Do something now that we know we have at least the first line of an
    HTTP transaction.
  else
    Good thing we didn't try to parse this service!
  end

Please excuse any errors in the above pseudo code - As you might be able
to tell I haven't implemented or tested them. :)

(BTW I really like the fact that lua has multiple return values! This is
always very convenient and is one thing that most languages lack. Yes,
it's features like multiple-value-bind that common lisp and lua programmers
take for granted but often find missing in other languages.)





It's great that you use the script's 'id values to interact with the C
code in output.cc:

char* formatScriptOutput(struct script_scan_result ssr) {
  ...
        result += "|  " + std::string(ssr.id) + ":" + sep;

But why stop there?

I can't seem to find any use of the 'description values in the C source or
any of the lua scripts. Does the description serve any purpose other than
being a comment in the source code?

Auto-documentation like this is a very powerful concept and could perhaps
be developed further. Consider quickly being able to see and choose exactly
which scripts will be applied against a discovered service. This would be
easy if most scripts used the "portrule can be string or function" suggestion
above. It isn't that big a deal right now but I can envision it becoming a
problem once we reach some critical threshold number of lua scripts in
the distribution!

In a program I started work on a while ago, which I call nuff, I make
extensive use of scheme macros and this sort of auto-documentation concept so
that a utility can automatically be well-documented. The parameters, their types
and acceptable values, whether this utility requires root privileges, etc etc,
can all be automatically determined and documented. I hope to someday muster
up the time and energy to complete this set of macros for publication.



Out of curiosity, I'm wondering why you chose NSE for some of your scripts
instead of version detection. For example, malware/kibuvDetection.lua seems
to accomplish the same thing that adding a match line or 2 to the probes file
would.



Anyways, these are just some random ideas. Take them for what they're worth.

Best,

Doug



_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev