Nmap Development: [NSE] Lua 5.2 Upgrade

[Patrick Donnelly <batrick () batbytes com>]

Hello list,

Over the past few months I've been working on a branch that upgrades
NSE to use Lua 5.2. I'll start off by giving you the location of the
branch so you can try it out:

$ svn checkout --username guest --password ""
https://svn.nmap.org/nmap-exp/patrick/nse-lua52 nse-lua52
$ cd nse-lua52 && ./configure --with-liblua=included && make
$ ./nmap --datadir . --script safe localhost

If you want to play with the Lua interpreter/compiler, you can simply
compile (after making nmap) using:

$ gcc -o liblua/lua liblua/lua.c liblua/liblua.a -lm -ldl
$ gcc -o liblua/luac liblua/luac.c liblua/liblua.a -lm -ldl

This upgrade brings numerous improvements. I've added a page to the
wiki which lists the most relevant or important changes to script
developers:

https://secwiki.org/w/Nmap/Lua_5.2

I've included the bulk of that wiki page at the end of this email for
archival purposes.


Please try out the branch and look at the wiki page to get a feel for
the changes. All feedback including "it works" is welcome! I would
like feedback on socket:receive_buf in particular as the code is new
but functionally the same.

The nse_bitlib.cc code is deprecated for now (the 'bit' library).
There may be some functional differences of significance between the
old bit library and the new Lua 5.2 bit32 library. I feel this should
be sorted out separate from this branch.


== Wiki page as Plain Text ==

New features in Lua 5.2 of interest to script developers:

o New hexadecimal escape sequence for strings. For example, instead of
using decimal escape "\010" you can instead use "\x0a".

o Lexical environments with _ENV [1]. Environments are now simply
upvalues of a function and you may introduce a new environment into a
scope by creating a new local named _ENV with a lifetime that expires
at the end of the scope. For example:

local print = print
function foo ()
  a = 5
  do
    local _ENV = {}
    print(a) --> nil
    a = 6
    print(a) --> 6
  end
  print(a) --> 5
end

You may also do things like:

local print = print
function foo (_ENV)
  print(a)
end
foo({a = 5}) --> 5
foo({a = 6}) --> 6

The old Lua 5.1 environments are completely gone. setfenv/getfenv are
gone as a result.

o Coroutines may yield across most C call boundaries. One of the
simplest examples of such an error in 5.1:

$ lua51
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
> F = coroutine.wrap(function(...)
> >   return pcall(function(...)
> >     return coroutine.yield(...)
> >   end, ...)
> > end)
> print(F("Hello"))
false   attempt to yield across metamethod/C-call boundary

This changes allows for coroutines in generic for loop iterators which
yield outside of the for loop. For example:

local function lines (socket)
  while true do coroutine.yield(socket:receive_buf("\r?\n", false)) end
end
for line in lines(socket) do
  --> do something
end

[The above example doesn't handle EOF or other errors.]

o Lua 5.2 comes with a default bit library called "bit32".

o Lua closures are now cached. The ramifications are best illustrated:

$ lua51
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
> for i = 1, 2 do print(function() end) end
function: 0x2104ad0
function: 0x2104fe0

$ lua52
Lua 5.2.0  Copyright (C) 1994-2011 Lua.org, PUC-Rio
> for i = 1, 2 do print(function() end) end
function: 0x1d51070
function: 0x1d51070

Notice the closures are equivalent (and we could confirm equality
using ==). Notice though:

$ lua52
Lua 5.2.0  Copyright (C) 1994-2011 Lua.org, PUC-Rio
> for i = 1, 2 do print(function() return i end) end
function: 0x796070
function: 0x796d30

This produces distinct closures because the upvalues are different.
Namely, the i local is recreated on each iteration of the loop so each
closure references a different i.

It is an optimization to cache closures. This optimization lessens the
performance impact of an anonymous function used in a loop. For
example:

-- Escape non-graphical characters
function escape (str)
  return (str:gsub("[^%w%s%p]", function (a) return
("\\x%02x"):format(a:byte()) end));
end

In Lua 5.1, if the above escape function is called in a loop, it would
create a closure for the call to gsub each time. These become
"garbage" because they are only used once. The way to correct this is
to pull the function out into a local above the escape function:

-- Lua 5.1 optimization for escape function
do
  local function helper (a)
    return ("\\x%02x"):format(a:byte())
  end
  function escape (str)
    return (str:gsub("[^%w%s%p]", helper));
  end
end

In Lua 5.2, this is largely unnecessary as the closure is cached for future use.

o Lua 5.2 now supports goto. Be sure the Raptor fences aren't out before use.

::forever::
goto forever

The goto mechanism was added partially to solve the lack of a continue
statement and nested break.

for i = 1, 10 do
  if i % 2 == 0 then goto cont end
  ::cont::
end

for i = 1, 10 do
  for j  = 1, 10 do
    for k = 1, 10 do
      if i + j + k == 99 then goto done end
    end
  end
end
::done::

o __pairs and __ipairs metamethods. You can now define how an object
is iterated over using the pairs and ipairs functions.

$ lua52
Lua 5.2.0  Copyright (C) 1994-2011 Lua.org, PUC-Rio
> t = {1, 2, 3, a = 1}
> setmetatable(t, {__ipairs = pairs}) -- make ipairs work like pairs
> for i, v in ipairs(t) do print(i, v) end
1       1
2       2
3       3
a       1


[1] http://www.lua.org/manual/5.2/manual.html#2.2

-- 
- Patrick Donnelly
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/