Home page logo

nmap-dev logo Nmap Development mailing list archives

Re: [Nsock] Scalable event handling
From: David Fifield <david () bamsoftware com>
Date: Thu, 25 Jul 2013 11:24:16 -0700

On Sat, Jun 15, 2013 at 09:13:46AM -0400, Henri Doreau wrote:
Longer version:
I'm working on cleaning and optimizing nsock, for the fun of it and
because of the plans we have to add features (like proper ssl proxy
support) and to eventually use nsock for connect scan. I'd eventually
like to introduce a hooking framework into nsock, so that both proxy
handlers and SSL code could transparently intercept and overload
connect/read/write operations. Overloading would consist in adding
events internally.

The problem is that adding many events is currently quite expensive.
You can easily observe it by stressing nsock a bit, via nping[1] for
instance. The reason why it's that slow is because each iteration of
the runloop goes through _all_ events and check for completion (I/O or

This big patch (it'd be complicated to split it...) I'm proposing here
makes nsock iterate over the active events only. IOW only IODs that
either have I/Os to handle or expired events are processed at each
iteration. I had to change the internal data structures for that:

  * Introduced a gh_heap data structure, to efficiently make priority
queues. All "expirables" (timers and timeouts) are now handled via
such a tree. So that we don't need to iterate over _all_ elements at
each nsock loop, only the active I/O events and the expired timers and
I/O events are processed. Needless to say that the difference is
noticeable when you have tens of thousands of events queued!

  * Replaced gh_lists containers by embedded objects (intrusive data
structure). All operations are inline, and adding an item to a list
doesn't allocate extra memory. It's needed so that we can manipulate
the events lists simply from events that we extract from the gh_heap.

Thanks Henri. This looks like a high-quality, worthwhile change as
always. I tried it and the TCP flood against localhost goes faster as
you say. gprof shows that epoll_loop stops dominating the profile after
the change.

+ #if !defined(container_of)
+ #define container_of(ptr, type, member) \
+         ((type *)((char *)(ptr)-(char *)(&((type *)0)->member)))
+ #endif

Perhaps this should use offsetof? offsetof appears to already be used in
the code of a few dependencies.

+   gh_lnode_t nodeq;
+   gh_lnode_t nodeq2; /* used for pcap events */

Can you elaborate on how nodeq and nodeq2 work? From what I understand,
nodeq2 is used for pcap-related events and nodeq is used for other kinds
of events. Why do these need to be separate? I wonder if there is a more
orthogonal way to do it. first_ev_next taking a nodeq2 parameter to
choose which list to work on is a little strange.

run_tests.sh is failing for me:

started UDP listener on port 55234 (pid 5694)
started TCP listener on port 55235 (pid 5696)
started TCP SSL listener on port 55236 (pid 5697)
nsock pool user data                            [OK]
test timer operations                           [OK]
set standard log levels                         [OK]
check error log levels                          [OK]
simple tcp connection                           [FAILED] (Connection refused)

Patch generated via git, let me know if you prefer another format or
if you want me to open a dedicated branch on nmap-exp.

It looks like a "git show" patch; "git format-patch HEAD^" is better
because it's easier to apply with "git am".

David Fifield
Sent through the dev mailing list
Archived at http://seclists.org/nmap-dev/

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]