Re: Call for testers: nsock-engines experimental branch

[Luis Martín García <luis.martingarcia () gmail com>]

On 07/09/2011 05:09 PM, Henri Doreau wrote:
> Hello,
>
> I have been working on an experimental nsock branch these days. My aim was to
> introduce modern and efficient IO management interfaces (like epoll) within
> nsock. A working version is now available and I would like to get some feedback
> about it.
>
>
> Development was made in three steps:
>
>   * Changed the internal architecture of nsock so that a nsock_pool doesn't
>   contains the complete lists of events for every IOD (nsock "supersockets")
>   anymore, but only the list of the active IODs instead. Each IOD now stores its
>   own event lists. This change was necessary given how the functions I wanted to
>   introduce work (see David's explanations[1]).
>
>   * Introduced a new layer at the lowest level (the "engines").
>   An engine is a set of functions, exporting a stricly defined interface to the
>   rest of the library for events polling. The first one I have "implemented"
>   was of course the select(2)-based one. It is mostly a refactoring of
>   existing nsock code. It is intended to become the fallback engine, in case no
>   more efficient one is available.
>
>   * Wrote an epoll(7)-based engine. The epoll system is linux specific (from
>   Linux 2.5.44 according to the man page) but should provide better performances
>   than select.
>
>
> This design allows to easily extend the library with highly efficient IO
> management systems (even the least portable ones) without breaking portability,
> as we still have a fallback engine. The one-engine-per-file implementation keeps
> the code modular.
>
> I have tested it as much as possible and noticed no regression so far. I would
> especially like to get feedback about large scale scans. The epoll engine
> removes the parallelism limitation we have with select (which cannot deal with
> more than FD_SETSIZE fds).
>
>
> --- HOW TO TEST ---
>   1) checkout the nmap-exp/henri/nmap-nsengines branch
>
>   2) call autoconf from nmap-nsengines/nsock/src to generate the new configure
>   file (which will now check for the presence of the epoll_* functions).
>
>   3) ./configure && make from within nmap-nsengines to build nmap and the
>   companion tools
>
>   4) run nmap -sVC, nping, ncat
>
>
> As I said the epoll engine will be automatically selected if it's available on
> your system (see nsock/src/nsock_engines.c). Even if you don't have epoll,
> testing would be very appreciated, to ensure that the select engine doesn't
> introduce any regression.
>
> There is still a lot of room for improvement (like code optimization or addition
> of new engines). Please note that the Visual Studio project file has not been
> updated on the branch. I would be glad if a windows user could do so (and tell
> me if I broke something on windows!). :)
>
>
> Regards.
>
> [1] http://seclists.org/nmap-dev/2011/q1/812
>
> --

Hi Henri,

I had a chance to test it and I'm happy to report that I worked as
expected. I am running Linux 2.6.35 and the configure script correctly
detected the availability of epoll(). Here's the relevant output.


[...]
checking for gcc option to accept ISO C89... none needed
checking for epoll_create... yes
checking for epoll_ctl... yes
checking for epoll_wait... yes
checking for gcc... (cached) gcc
[...]


I tested nmap and nping and, as I said, they work as expected.

I'm not sure this is relevant but I've noticed that Nping finishes a bit
later when I run it with the new nsock than with the old. I attach the
output of three different runs for each version. Files named newX.txt
correspond to the new nsock, files named oldX.txt correspond to the old.

Some more tests:

[+] New Nsock
time sudo ./nping localhost --tcp -p 22 --count 10000 --rate 10000
real    0m5.818s
user    0m0.650s
sys    0m0.350s
time sudo ./nping localhost --tcp -p 22 --count 10000 --rate 10000
real    0m5.959s
user    0m0.560s
sys    0m0.410s
time sudo ./nping localhost --tcp -p 22 --count 10000 --rate 10000
real    0m5.889s
user    0m0.710s
sys    0m0.280s

[+] Old Nsock
time sudo nping localhost --tcp -p 22 --count 10000 --rate 10000
real    0m5.796s
user    0m0.790s
sys    0m0.200s
time sudo nping localhost --tcp -p 22 --count 10000 --rate 10000
real    0m5.978s
user    0m0.610s
sys    0m0.320s
time sudo nping localhost --tcp -p 22 --count 10000 --rate 10000
real    0m5.833s
user    0m0.720s
sys    0m0.240s

Forget about it because these last tests show that the difference is not
significant at all. The differences in Nping's output can be due to the
way it handles events and timers (something that could certainly be
improved), and the differences introduced in the new version of nsock.
As both versions result in similar runtimes, I think this matter should
be completely ignored. (Certainly, I could have erased half of this
email, but since I took the time to write it, I though I'd leave it for
future reference ;)


Regards,

Luis MartinGarcia.

Attachment: new1.txt
Description:

Attachment: new2.txt
Description:

Attachment: new3.txt
Description:

Attachment: old1.txt
Description:

Attachment: old2.txt
Description:

Attachment: old3.txt
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/