ultra_scan-based host discovery now completed; testing needed

[David Fifield <david () bamsoftware com>]

Hi,

The changes to Nmap that cause it to use ultra_scan for host discovery
instead of massping are now completed. Here's the announcement of the
prototype for some background:

        http://seclists.org/nmap-dev/2007/q3/0076.html

The changes are not in the nmap or soc07 Subversion branches. They are
in their own branch, which you can check out with

        svn co svn://svn.insecure.org/nmap-exp/david/nmap-massping-migration

When I say that the work is completed, I mean that the programming work
that makes host discovery use ultra_scan is done. Testing and tuning are
needed to make it work as well as the old massping. The new code is
slower and detects fewer hosts than the old code. I need your help to
figure out the reasons why that is and fix them.

I ran some host discovery tests against lists of random IP addresses
generated by commands like

        nmap -n -sL -iR 50 | grep '^Host ' | awk '{ print $2 }' > random-hosts-50

Here are the results against 50 hosts. In the command lines, "nmap" uses
massping, while "nmap-massping-migration/nmap" uses the new ultra_scan
host discovery.

        # nmap -sP -PS -PE -n -iL random-hosts-50 -oN old-massping-50
        Nmap finished: 50 IP addresses (3 hosts up) scanned in 8.685 seconds

        # nmap-massping-migration/nmap -sP -PS -PE -n -iL random-hosts-50 -oN new-massping-50
        Nmap finished: 50 IP addresses (3 hosts up) scanned in 19.464 seconds

It looks good, except that it's much slower. Here are the results
against 500 hosts:

        # nmap -sP -PS -PE -n -iL random-hosts-500 -oN old-massping-500
        Nmap finished: 500 IP addresses (37 hosts up) scanned in 44.430 seconds

        # nmap-massping-migration/nmap -sP -PS -PE -v -n -iL random-hosts-500 -oN new-massping-500
        Nmap finished: 500 IP addresses (34 hosts up) scanned in 71.089 seconds

This is worse. Not only is the new code slower, it misses some hosts
that were detected before.

Finally, here are the results against 50000 hosts.

        # nmap -sP -PS -PE -n -T4 -iL random-hosts-50000 -oN old-massping-50000
        Nmap finished: 50000 IP addresses (2641 hosts up) scanned in 4195.506 seconds

        # nmap-massping-migration/nmap -sP -PS -PE -n -T4 -iL random-hosts-50000 -oN old-massping-50000
        Nmap finished: 50000 IP addresses (2391 hosts up) scanned in 24828.195 seconds

Here the new code is about five times slower and misses 250 hosts (about
10% of them).

I feel out of my depth in figuring this out alone. I think the slowness
is caused mainly by a reduced packet sending rate. I think the missed
hosts are due partly to ultra_scan giving up on hosts too soon. I saw a
few cases where a host's reply came back in about 30 seconds, after the
probes to that host had expired.

I wrote a script called host-list-compare.py to help find the problems.
It's in Subversion with the rest. Run it like this:

        # ./host-list-compare.py old-massping-500 new-massping-500
        37 hosts in old-massping-500.
        34 hosts in new-massping-500.

        3 extra hosts in old-massping-500:
                201.242.55.103
                84.152.100.142
                209.255.3.246

This can help you know which hosts to focus on if you're looking at this
with a protocol analyzer.

David

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org