Home page logo

nmap-dev logo Nmap Development mailing list archives

Re: [NSE] Child Coroutine Patch with Explanation
From: Patrick Donnelly <batrick () batbytes com>
Date: Thu, 4 Jun 2009 18:08:59 -0600

Hi Fyodor,

On Thu, Jun 4, 2009 at 5:57 PM, Fyodor<fyodor () insecure org> wrote:
On Thu, Jun 04, 2009 at 05:46:53PM -0600, Patrick Donnelly wrote:
This thread attempts to answer a problem that came up on the NSE IRC
meeting. We are attempting to solve the issue of a script using
coroutines that make blocking calls on network objects. This e-mail
attempts to give a detailed overview of the problem, some potential
solutions, and example use cases for a proposed solution.

Hi Patrick.  Thanks for the detailed write up.  Regarding the
nse.new_thread proposal ... what would be your recommendation for
communicating status and data between the worker threads or between
them and the parent?

The function passed to nse.new_thread could (would) have upvalues
shared with the host thread's functions it can modify as it runs to
communicate status or results. The worker threads may also use the
environment of the script which created them to communicate results.
Here is an example of the "sharing upvalues":

function action (host)
  local status;
  local results;
  local function worker_main (host)
    -- do stuff
    results = "some value";
    status = "DONE";
  while status ~= "DONE" do
    stdnse.sleep(1000); -- sleep 1 second
  -- do stuff with results

 Am I correct that, at the low levels, they won't
actually be running concurrently even on multi-processor or multi-core
systems?  So can they safely share global variables without needing
any sort of synchronization primitives?

These extra threads would appear to be separate scripts running. The
difference is we engineer the thread to not return any results (so we
do not attempt to report anything) and any debug output (errors) is
squelched. So, these aren't OS level threads. (I know this terminology
can be confusing but I try to follow Lua's guidelines here).

A running thread can always assume that all of its changes to globals,
locals, etc. are done synchronously with respect to other threads.

 And for the cases where they
do need synchronization, can the threads individually use our mutexes?

Absolutely! However, there does need to be some mechanism of
synchronization for scripts with their worker threads other than
mutexes. The issue with mutexes is we cannot unconditionally wait
until another thread wakes us up (except through something like
stdnse.sleep with a loop). Ideally, we would have some condition
variable similar to POSIX threads that would allow a script to wait
until a worker has new information to communicate. There would need to
be deadlock protections in place such as associating a condition
variable with a group of worker threads so that if a worker ends we
force the script to wake up. This protection would be similar to what
is now in place for mutexes.

Also, does the new_thread proposal depend on the nse_yield patch, or
are they independent issues?

Completely independent.

-Patrick Donnelly

"Let all men know thee, but no man know thee thoroughly: Men freely
ford that see the shallows."

- Benjamin Franklin

Sent through the nmap-dev mailing list
Archived at http://SecLists.Org

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]