Re: [NSE] Child Coroutine Patch with Explanation

[Patrick Donnelly <batrick () batbytes com>]

Hi Fyodor,

On Thu, Jun 4, 2009 at 5:57 PM, Fyodor<fyodor () insecure org> wrote:
> On Thu, Jun 04, 2009 at 05:46:53PM -0600, Patrick Donnelly wrote:
> > This thread attempts to answer a problem that came up on the NSE IRC
> > meeting. We are attempting to solve the issue of a script using
> > coroutines that make blocking calls on network objects. This e-mail
> > attempts to give a detailed overview of the problem, some potential
> > solutions, and example use cases for a proposed solution.
>
> Hi Patrick.  Thanks for the detailed write up.  Regarding the
> nse.new_thread proposal ... what would be your recommendation for
> communicating status and data between the worker threads or between
> them and the parent?

The function passed to nse.new_thread could (would) have upvalues
shared with the host thread's functions it can modify as it runs to
communicate status or results. The worker threads may also use the
environment of the script which created them to communicate results.
Here is an example of the "sharing upvalues":

function action (host)
  local status;
  local results;
  local function worker_main (host)
    -- do stuff
    results = "some value";
    status = "DONE";
  end
  while status ~= "DONE" do
    stdnse.sleep(1000); -- sleep 1 second
  end
  -- do stuff with results
end

>  Am I correct that, at the low levels, they won't
> actually be running concurrently even on multi-processor or multi-core
> systems?  So can they safely share global variables without needing
> any sort of synchronization primitives?

These extra threads would appear to be separate scripts running. The
difference is we engineer the thread to not return any results (so we
do not attempt to report anything) and any debug output (errors) is
squelched. So, these aren't OS level threads. (I know this terminology
can be confusing but I try to follow Lua's guidelines here).

A running thread can always assume that all of its changes to globals,
locals, etc. are done synchronously with respect to other threads.

>  And for the cases where they
> do need synchronization, can the threads individually use our mutexes?

Absolutely! However, there does need to be some mechanism of
synchronization for scripts with their worker threads other than
mutexes. The issue with mutexes is we cannot unconditionally wait
until another thread wakes us up (except through something like
stdnse.sleep with a loop). Ideally, we would have some condition
variable similar to POSIX threads that would allow a script to wait
until a worker has new information to communicate. There would need to
be deadlock protections in place such as associating a condition
variable with a group of worker threads so that if a worker ends we
force the script to wake up. This protection would be similar to what
is now in place for mutexes.

> Also, does the new_thread proposal depend on the nse_yield patch, or
> are they independent issues?

Completely independent.

-- 
-Patrick Donnelly

"Let all men know thee, but no man know thee thoroughly: Men freely
ford that see the shallows."

- Benjamin Franklin

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org