Home page logo

nmap-dev logo Nmap Development mailing list archives

Fwd: SOC idea
From: Jason Gerfen <jason.gerfen () gmail com>
Date: Fri, 10 Jan 2014 05:48:00 -0700

---------- Forwarded message ----------
From: Jason Gerfen <jason.gerfen () gmail com>
Date: Thu, Jan 9, 2014 at 5:31 PM
Subject: Re: SOC idea
To: Jacek Wielemborek <d33tah () gmail com>

Thanks for the response!

On Thu, Jan 9, 2014 at 10:21 AM, Jacek Wielemborek <d33tah () gmail com> wrote:

See my answer inline below.

09/01/2014 07:40:35 Jason Gerfen <jason.gerfen () gmail com>:
I just posted on the announce list, which led me to
http://nmap.org/soc/#creative and now this message.

They say you miss 100% of the shots you never take so here it goes...

Congratulations, great attitude! I understand that you're interested in Google
Summer of Code. If so, getting one's idea accepted is definitely not that
difficult if you're motivated (I, for one, am a good example :)). You already
did a few things right - got involved early, came up with an idea and
requested comments on the mailing list. So far, way to go, IMHO :)


I have already started a project which utilizes the current nmap
binaries and improves the speed dramatically. The project has only
recently started and with some preliminary benchmarks shaves 20seconds
off simple host discovery scans.

Details of the project can be found at
https://github.com/jas-/node-libnmap &


I took a quick look at your code and if I understand it correctly, it's
currently limited to host discovery. There's not much written yet, but I
already have a few potentially interesting comments:

1. Your Github readme suggests that you might be using Nmap 5.51. If this is
the case, do upgrade to the latest version before benchmarking your code,
otherwise you might be wasting time.

I am currently using and testing against 5.51, so green there.

2. Consider using -oX instead of -oG. This way you will get an ouput format
less likely to be changed in the future. As far as I know, -oG is meant for
being read by a human, not parsed by a machine.

Am I wrong to assume -oG will allow the following?

$ nmap -sn -oG - | awk 'up { print }'

Really due to the nature of the language, in this case node.js which
relies heavily upon Objects, and more specifically JSON formatted
objects a simple regex combined with a capture group of the results is
going to be faster than traversing XML elements.

3. In addition to using -sn, you might also want to call -n, which disables
rDNS queries. This might speed up the queries to a greater extent.

This change has already been applied, thanks for that I did not see
that in the docs, but may have over looked it.

4. See if you're running Nmap in priviledged mode (one, though insecure, way
to enable it is to run Nmap as root. For better solutions, see
https://secwiki.org/w/Running_nmap_as_an_unprivileged_user). This enables many
speedups, such as - in your discovery case - ARP ping, which is both more
accurate and faster in case of local networks.

While I do appreciate the information about this type of setup,
because the project really should work as a 'plug and play' turn key
solution for simple scanning purposes this would be left to the sys
admin and can be applied with the optional flags a developer can use
with the project.

It might also be a good idea to do some research on cutting down the Nmap
initialization time - for example by compiling it with a C library lighter
than glibc.

This again is something that will be left to the sys admin in this
projects case.

Also, consider the use case you're planning to eventually accomodate. If your
goal is to speed things up, there are things that you can do faster by
spawning multiple Nmap processes than by running a single process. This
usually not due to SMP though - currently, AFAIK, the biggest problem is that
Nmap scanning is split into phases. There's the discovery phase, rDNS phase,
port scanning phase, NSE, traceroute, OS discovery, service discovery etc...
If in any of the hosts lags behind in any of the phases, all remaining ones
have to wait. Currently everybody walks around the problem by scripting stuff
like you do, but we all know that it's more or less a temporary solution.

Yeah, I can see why it is easier to split the subnet, range etc into
chunks and create individual process for each block of address to help
optimize each processor core.

There are other use cases for your library if you keep developing it - some
things cannot currently be done with pure command line and it's possible that
not all of them will ever be added to Nmap. One of the features missing in the
Nmap core that we might expect to be added to Nmap soon is port specification
that would allow to exclude certain ports. You could probably implement it in
your library more easily than you would do that in the C code (though it's not
the correct solution actually).

Good to know.

Another example is a feature that I once needed, but I would be surprised to
ever find in Nmap core. I needed to perform a lot of OS discoveries on hosts
whose closed/open TCP/UDP ports differed. I wrote a script myself that
performed the probing and I believe that there are many use cases like this
that could be made easier with a cool node.js interface. Keep in mind though
that if you plan to go around problems that way, you have to solve many
problems that Nmap takes care of for you.

Haven't run into any yet, but I am sure as the project progresses I
might. I have been looking at the py-libnode implementation as a sort
of guide except I am focusing this project on optimizing the scans
using node.js non-blocking IO functionality combined with separate
processes for each subnet chunk.

One of the problems is timing. When I probed my first few thousands of hosts, I
already got two automated abuse complaints from some trigger-happy IDSes. I
quickly understood that even though I actually sent SYNs to just three ports,
it summed up to numbers big enough for these systems to complain. I then wrote
a script that ordered the targets so that it's less likely for two hosts from
the same network to be probed in a short period of time. That's one of the
things you might have to do for your tool to be useful in larger scans.

Never considered this, as most IDS are at perimeters in our env.

My final advice for you is that you really should read Fyodor's book. It's an
amazing guide to Nmap's functionality, with a lot of explanations that will
help you understand what actually happens and when will you need the features.
It's a few hours of reading, but I guarantee you that your knowledge of Nmap
will greatly increase, which you're bound make use of in the future :)
Especially that the default "nmap scanme.nmap.org" turns on quite a lot of
options and many of them are very risky if you want to stay stealthy.

This is definitely good information, didn't know there was a book.

Whoa, quite a lengthy post. I hope I helped and didn't discourage you. Keep up
your work, it might actually prove useful if you think it over!

Thank you again sir, much appreciated.

Jacek Wielemborek
Sent through the dev mailing list
Archived at http://seclists.org/nmap-dev/
Sent through the dev mailing list
Archived at http://seclists.org/nmap-dev/

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]