mailing list archives
Re: Ncat + Lua - GSOC feedback request
From: Henri Doreau <henri.doreau () gmail com>
Date: Sun, 16 Jun 2013 23:01:51 +0200
A couple comments inline.
2013/6/16 Jacek Wielemborek <wielemborekj1 () gmail com>:
On Fri, June 14, 2013 at 10:43:42 David Fifield wrote:
On Sun, May 12, 2013 at 05:51:27PM +0200, Jacek Wielemborek wrote:
2. Ideas for the use cases
So, we've got this well-documented shiny tool that speaks Lua now.
Apart from all the joy of starting a script with #!/usr/bin/ncat -L
header line, what else is there? I already mentioned: 1) NSE protocol
libraries being used for more personal needs and 2) better NSE testing
Here are my three use cases. May I suggest that you create a SecWiki
page containing a list of use cases?
Unfortunately I can't. I reported my problems here:
lua-exec option. This is perhaps the least ambitious, least
controversial, and easiest notion of Lua integration. I think you should
do it first before anything else. What it means is that you can do
ncat --lua-exec program.lua
Ncat will fork and exec program.lua as if it were a shell script with
--sh-exec. In fact, the above is the same as
ncat --sh-exec "lua program.lua"
except that you don't need a separate installation of Lua. (And also,
maybe Ncat's Lua interpreter will have libraries available by default or
I like this idea as a starting point - I understand that all the the calls to
io.write()/print() should direct to the socket instead of the stdout and all
calls to io.read() should read from the string? (oh, by the way, is the
forking really necessary?)
+1 for this as a starting point. As there's no exec(), forking is not
striclty necessary but it's a way to easily handle multiple
connections. I believe this first prototype should be kept simple.
Implement telnet negotiation and CRLF replacement. The -t and -C options
are implemented as special-purpose C code. (Grep for "dotelnet" and
"fix_line_endings".) It would be nice if these along with other things
could be implemented as little Lua snippets. What I'm picturing here is
not separate script files, but actual Lua source code as strings within
the Ncat source code. Each little program would be like a filter that
does CRLF substitution or whatever. This use case seems straightforward
but I think the implementation is not trivial (but neither is it super
hard). CRLF substitution needs to keep state across calls in case a CRLF
crosses a buffer boundary. Telnet negotiation needs to send data
in-band in response to information it receives. So neither is a simple
block-at-a-time byte filter.
I'll look into that, at the moment I don't really know much about the
negotiation protocol (but I'll catch up). I'm a bit worried about hardcoding
Lua code in the Ncat core though - what if the user wants to drop Lua
functionality? Fallback to C routines? Or should we make Lua a requirement for
Ncat to compile?
Try to keep lua-related things together and between preprocessor
guards to let users choose whether to compile w/ or w/o lua. It
shouldn't be a requirement for ncat to compile. See nmap (grep for
NOLUA) to see how this can be done.
WebSocket mode. See RFC 6455. It would be nice to do
ncat --websocket ws://example.com:8000/
ncat --listen --websocket ws://localhost:8000/
What this involves is first sending a special HTTP request asking for a
protocol change to WebSocket. Then you have to receive a response and do
some SHA1 calculations and do error handling. After that you can send
and receive data on the same TCP connection, but not directly: each
little block of data needs to be encapsulated in WebSocket messages and
frames. This is quite a heavyweight use case. An implementation that can
do it, can probably do most of what we would want Lua integration to do.
You should also be able to use SSL WebSocket in the obvious way:
ncat --websocket wss://example.com:8000/
I love this one - it would add some really new functionality to Ncat and be a
great demo of what the Lua engine could (and should) be be capable of. The
only thing that I don't exactly understand is the usage - do you mean that
every Lua script should be capable of messing with argv interpretation (sounds
dangerous) or WebSocket should be a special script? Option three that comes to
my mind is invocation like:
ncat --lua-exec "websocket,url=ws://example.com:8000"
ncat --lua-exec "websocket" example.com 8000
ncat --lua-exec "websocket" --ssl example.com 8000
ncat --lua-exec "websocket,url=ws://example.com:8000,mode=listen"
ncat --lua-exec "websocket" -l -p 8000
These examples raise questions about how much of the the Ncat internal
interface should be exposed to the user - my initial plans suggested to allow
the scripts to read (and perhaps change) the "options" structure, here I also
introduce a special script parameter list. That's probably going to be quite
some work for me to do.
Here again, you can have a look at NSE, but if such websocket code is
implemented within ncat's core, you shouldn't have too much problem.
The C code can access everything and pass whatever is needed to lua.
Chat mode for me is pretty much meh. If it's easy to do with some inline
Lua, fine. I don't think it's worth adding code to allow custom
nicknames or anything fancy.
I pretty much considered it a nice server demo - from the beginning I assumed
that Ncat-Lua should cover all the Ncat's functionality, but when I now think
of it, perhaps it's better to focus on the client side first.
I don't find the use of NSE libraries compelling. You'll have to
convince me. I think I don't yet understand the better NSE testing idea.
(I'll reply to that below, where you answered to the section 4 of my original
e-mail, which is also about NSE)
3. Hard requirements
* Quite a lot of planning
I definitely consider it a requirement number one for this task. I
don't really believe that it's possible to integrate Lua with Ncat
properly without prior planning, especially if we're heading for NSE
compatibility. The planning phase is when I expect to need my mentor
the most, so that the code would be written well and once, with no
further rewriting ever needed.
If I may offer a different perspective, it may be that lots of
pre-planning (like http://c2.com/cgi/wiki?BigDesignUpFront) isn't right
for this job. Partly because we don't even have a list of requirements
yet, and only a few use cases. You would be trying to design something
that is under-specified. I am fine with building some prototypes and
throwing away some designs.
Lots of planning up front also increases the risk that there will be
nothing useful at the end of the summer. I suggest you implement
--lua-exec first, and from there you will have a better idea of the
Hm, we already talked about this on IRC and I agree that might be the better
option. So, instead of strictly following the timeline I wrote for my
proposal, I'll be focusing on --lua-exec for now.
4. Open questions
How much compatibility with NSE is possible for Ncat without getting
I think that you should not aim for NSE compatibility unless you already
have a worked-out design for it. What would it mean for Ncat to be
compatible with http-date.nse, for example? With http.lua? I don't know
what that even means. I fear that without a design or use cases in mind,
we'll have an implementation that won't be used.
Honestly, I don't have any design for the NSE part (and the IRC log from my
last e-mail suggests it would be a really difficult task). My idea of
compability with http-date.nse meant that the user runs something like:
ncat --script http-date localhost 80
And the action method would be run against the server. The compability with
http.lua would mean that it would be possible to use the library inside the
I wouldn't mind dropping NSE support for now though; it was just an idea I
meant to pass to you guys and find out if I it's possible to do in three months
along with a Lua interface to the rest of Ncat's functionality (with some
mentoring of course).
If I understand correctly, that would mean adding quite some code to
ncat just to have an alias to
"nmap --script http-date -p 80 localhost". NAK for me on that one...
Sent through the dev mailing list
Archived at http://seclists.org/nmap-dev/