Re: massping-migration and other dev testing results

[David Fifield <david () bamsoftware com>]

On Wed, Sep 12, 2007 at 02:07:19AM +0000, Brandon Enright wrote:
> On Tue, 11 Sep 2007 12:36:05 -0600 David Fifield <david () bamsoftware com> wrote:
> > Actually another thought just occurred to me. The
> > nmap-massping-migration branch also has debugging and profiling flags
> > turned on. I wonder if that could have such a large effect? I've noticed
> > that these flags make Nmap use a lot more CPU on my system, but I
> > haven't measured them to actually slow down any scans.
>
> I had noticed the additional CPU usage but I didn't think much of it.  I've
> compiled debug and profiling code before and not seen this kind of
> performance hit.  Perhaps profiling has a lot more work to do with C++ code?
>
> > Second, please run side-by-side the latest /nmap (the same as your test
> > above) and the latest nmap-massping-migration (r5824, the same as above
> > the exception of the removal of debugging flags). Run your 'a' tests
> > concurrently, then your 'b' tests concurrently.
>
> This made the difference we were looking for.

Groovy!

> I was under the impression that --randomize-hosts only randomizes host
> within a ping group.  The fact that it also increases the size of the
> ping group is *huge*.  The man doesn't make this all that clear.  I went
> ahead and looked at the code for this and I have a couple of thoughts.

You're right, it only randomizes hosts within a ping group. So it
wouldn't have any effect on those empty blocks, except, as you've said,
that it also increases the ping group size.

--randomize-hosts isn't in the nmap -h output either.

> First, PING_GROUP_SZ is set to 4096.  When you use --randomize-hosts
> 'o.ping_group_sz = PING_GROUP_SZ * 4;' is run.  The man says the group can
> grow to up to 8096.  There doesn't appear to be any special cap so the
> group size would actually be 16384.

Oops. We changed the default PING_GROUP_SZ from 2048 to 4096 a little
while ago and I didn't know that was in the reference guide. That's
fixed.

> Second, it really surprises me that an important value like this isn't
> adjustable.  I thought --min-hostgroup set the ping group size but after
> looking at the code, this doesn't appear to be the case.  I suppose most
> people aren't scanning 10k+ hosts so it doesn't matter much.  For those
> that do though, it really matters.

--min-hostgroup hasn't ever had an effect on ping scans, as far as I can
tell (not even with massping). Host discovery kind of does its own
thing, which is a legacy from massping.

> Since this value is already so large, using the value from --min-hostgroup
> is probably not a good idea.  Perhaps another option like --min-ping-group.

Hmm.

> I have preliminary results that show larger ping_groups (using either
> - --randomize-hosts, or recompiling) to really help.  Some of the scans are
> still going though so I'll have to send a follow up email illustrating this
> when I get home.
>
> Okay, here are scan results.  To recap, my 'a' scan uses -T5 and has a
> min/max parallelism of 1024 against ports 135,139,445,3389 on 180k IPs.  My
> 'b' scan just uses -T5.
>
> The 'a' scans were started simultaneously for the Nmap SVN trunk as well
> as the massping-migration branch (MPM).  The same goes for the 'b' scans.
>
>
> david_mpm_r5824a.nmap:
> # Nmap done at Tue Sep 11 20:54:22 2007 -- 186368 IP addresses (11790 hosts
> up) scanned in 288.591 seconds
>
> david_nmap_r5824a.nmap:
> # Nmap done at Tue Sep 11 20:54:22 2007 -- 186368 IP addresses (11936 hosts
> up) scanned in 287.305 seconds
>
> All I can say here is whoa, that's close. And fast.
>
>
> david_mpm_r5824b.nmap:
> # Nmap done at Wed Sep 12 00:17:31 2007 -- 186368 IP addresses (15628 hosts
> up) scanned in 2640.259 seconds
>
> david_nmap_r5824b.nmap:
> # Nmap done at Wed Sep 12 00:49:08 2007 -- 186368 IP addresses (15901 hosts
> up) scanned in 4536.876 seconds
>
> Okay, now your new code is starting to shine.  Much faster, almost as
> accurate.

There's a script called host-list-compare.py in the
nmap-massping-migration branch. It takes two .nmap log files and prints
out which hosts are up in the first that aren't in the second, and vice
versa. I'm curious to know whether the mpm runs just missed a bunch of
hosts, or they missed a bunch of hosts but found a bunch of others.
Please run that script on your logs for these most recent scans and
report back the two lines that say "N extra hosts in X.nmap".

And then, if you wouldn't mind, grep through the logs for a few of the
addresses that were missed and see what the differences are. It's
significant if, say, a response is received late (after the end of a
ping group, which --packet-trace would show) versus not received at all.

I don't know what the hosts on your network are like. During my testing
for the migration, I found that there are some weird hosts out there
that consistently take 30 seconds or more to respond to a probe. If
those 30 seconds don't elapse before the ping group is finished, the
host is marked down. There's not much you can do about those except use
a larger ping group or less aggressive timing parameters.

I'm not suggesting that's what's going on with your scans, it's just one
explanation I thought of. Another is that maybe the more robust
congestion window actually is oversaturating the line enough that
responses are being dropped without being detected.

> I'm going to follow up with tweaked PING_GROUP_SZ results but here is a
> preview. I ran david_mpm_r5824b.nmap (took 2640 seconds) with
> - --randomize-hosts and shaved off 600 seconds:
>
> david_mpm_r5824c.nmap:
> # Nmap done at Wed Sep 12 01:27:32 2007 -- 186368 IP addresses (13327 hosts
> up) scanned in 2019.837 seconds
>
> This scan did find 2k fewer hosts, but since they were done around 5pm
> local time some of this drop-off is hosts being turned off.

That's a nifty result. When we made the change to 4096 in ping groups, I
did some testing and didn't see much of a difference. I bet larger ping
groups have more of an effect on your network.

It looks like we're almost there with the migration. I've talked to
Fyodor, and I plan to merge this in the near future so that it can go
into a prerelease.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org