Re: [nmap-svn] r25684 - nmap-exp/calderon/nselib

[Djalal Harouni <tixxdz () opendz org>]

On Tue, Aug 09, 2011 at 11:46:53PM -0400, Patrick Donnelly wrote:
> Hi Paulino,
>
> On Tue, Aug 9, 2011 at 3:33 AM,  <commit-mailer () insecure org> wrote:
> > Author: paulino
> > Date: Tue Aug  9 00:33:52 2011
> > New Revision: 25684
> >
> > Log:
> > improvements
> >
> > Modified:
> >   nmap-exp/calderon/nselib/httpspider.lua
> >
> > Modified: nmap-exp/calderon/nselib/httpspider.lua
> > [...]
> > +local function add_visited_uri(uri, page_obj)
> > +  VISITED_MUTEX "lock"
> > +  if nmap.registry[HTTPSPIDER_DATAKEY]["visited"][uri] == nil then
> > +    nmap.registry[HTTPSPIDER_DATAKEY]["visited"][uri] = page_obj
> >   end
> > -  VL_MUTEX "done"
> > +  VISITED_MUTEX "done"
> >  end
>
> There's no need for this mutex here. Scripts (Lua threads --
> coroutines) do not yield asynchronously; that is, they do not yield at
> arbitrary locations in the code. Mutexes in NSE are used to serialize
> access to some resource *when the script may yield between the lock
> and unlock*.
What Patrick tries to say here is that if your _coroutines_ access
a share resource between some code that can yield then you need
to protect this block of code with mutexes.


In this function you should not use mutexes for one (or two reasons):

1) There are no calls to any function that can yield, if your code was:
  local function add_visited_uri(uri)
      VISITED_MUTEX "lock"
      if nmap.registry[...]["visited"][uri] == nil then
        local page_obj = socket:receive()
        nmap.registry[...]["visited"][uri] == page_obj
      end
      VISITED_MUTEX "done"
  end

Then you can use mutexes, since scripts or coroutines will all yield at
the socket:receive() call.

2) If there are calls to functions that can yield but you are sure that
you are not processing some shared resources. In your example you are
accessing the registry, however your code just fetches and stores page
objects in the registry (no extra complex operations), so if you are
absolutely sure that your workers will not pass the _same_ uri value to
this _local_ add_visited_uri() function then you can avoid mutexes even
if there are calls to functions that can yield. (this one is just to
clarify :) )

> Scripts only yield *explicitly* through calls to socket
> functions (e.g. socket:connect(...) or nmap.sleep(...)) or through the
> concurrency control mechanisms (mutexes and condition variables).
This is the list of functions that will make your code yield:
* condvar("wait")
* mutex("lock") -- will yield if the mutex is busy
* stdnse.sleep()

* socket:connect()
  - If too many threads have open sockets, we reach the max_parallelism
    20 or Nmap's max_parallelism option.
  - If the connection succeed.
* socket:reconnect_ssl()
* socket:send()
* socket:sendto()
* socket:receive()
* socket:receive_*()
* pcap:pcap_receive()

and coroutine.yield()


Patrick should we add them to the scripting.xml doc ?

-- 
tixxdz
http://opendz.org
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/