Re: Call for feedback: Nmap regression testing and the future of nmap-nsock-scan

[David Fifield <david () bamsoftware com>]

On Tue, Jul 22, 2014 at 10:44:26PM +0200, Jacek Wielemborek wrote:
> Two weeks ago I had a conversation with my Google Summer of Code mentor,
> David Fifield. During the meeting we agreed that my project (nsock-based
> -sT port scanning) is - for numerous reasons - impossible to complete by
> the end of GSoC. In order to prepare the code for merging, I would need
> to perform a few tasks that are very time-consuming, such as:

I like the way the nmap-nsock-scan branch is progressing. I hope that in
any case, you can leave off the work with a list of TODOs (like you've
done in your post). I think the advantages of Nsock-based scanning,
particularly proxy support, will be valuable to Nmap users.

I'm not too worried if some scans run slower. Some performance issues
will inevitably have to be shaken out through some actual use. I ran
into similar difficulties during my GSoC when I rewrote host discovery
using ultra_scan.
http://seclists.org/nmap-dev/2007/q3/140
https://www.bamsoftware.com/wiki/Nmap/HostDiscoveryBenchmarks

> As you can see, the amount of work that is left is well beyond the
> remaining four weeks of Google Summer of Code, which means that the
> feature won't be merged by the end of the program. As a result of that,
> David asked me if I could come up with something that would already be
> usable by that date. My regression testing script came to my mind, which
> already can automatically build various branches of Nmap under Linux,
> scan scanme.nmap.org using the resulting binaries, compare the results
> and e-mail a report. The report includes the commands that were
> executed, information about the timings and whether the port information
> was consistent or not (if not, a nicely-looking diff is generated so
> that you can quickly tell if it's the Linux ephemeral ports bug or
> something even more serious). In addition to that, a plot that shows
> both per-host and per-group active probe count and congestion control
> variables such as cwnd or ssthresh. The plot is attached to the e-mail
> and multiple recipients can be specified. I also created an environment
> that can be used to measure the relationship between packet drop
> percentage and Nmap accuracy which consists of two virtual machines, a
> script that executes commands over SSH and interprets their output and a
> Scapy script that SYN+ACKs every SYN packet so that any connection
> attempt that made it to the scanned host would come up as an open port.

I like the daily testing script. What I picture is the script running
once a day, mailing its reports to a mailing list or central log where
they can be stored and looked back at historically. Maybe it only emails
people directly when there's an error.

> * Integrate the tests involving other VMs: so far, the tests that rely
> on two VMs connected to each other are held in a separate script that
> needs to be called manually. I could integrate it and try writing a
> framework for tests that makes it easy to sniff for various packet
> properties necessary for a test case to succeed,

If you're using QEMU/KVM, you can get packet dumps with the "-net dump"
device. If you're using tcpdump now, that might be easier.

> * Test the program under other Unixes: my script currently relies on an
> Unix environment, but I hadn't yet tested if any GNUisms sneaked in. I
> could try running it under some BSD flavors and see if it works and if
> not, try correcting it,

This is good, but I wouldn't make it a first priority.

> * Add Windows support: building Nmap under Windows is completely
> different compared to any Unix system. Even if I relied on Cygwin (which
> is tempting, because it would be simpler), I would need to rewrite the
> building code to make it use Microsoft Visual Studio. I'm not convinced
> how useful that would be without a Windows server though,

A Windows tester would be very useful. But as you say, it's more
difficult to automate. I would put Windows support as a higher priority
than other Unixes, but it will also be more difficult.

David Fifield
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/