Re: New development in host discovery: response rate scaled congestion control

[David Fifield <david () bamsoftware com>]

On Thu, Sep 06, 2007 at 08:53:33PM +0000, Brandon Enright wrote:
> David Fifield <david () bamsoftware com> wrote:
> > You said that you ran all the scans at the same time. This new technique
> > ought to approximate TCP congestion control more closely--if four
> > concurrent Nmaps are run, each one should run about 1/4 as fast.
>
> This makes sense theoretically.  Practically though, this machine is on
> gigabit Ethernet and capable of sending hundreds of thousands of small
> packets a second.  I think the limiting factor in sending packets is the
> module for the NIC, OS buffers, and syscall servicing rate.
>
> This machine normally has a minimum of 16 nice'd Nmap running, each against
> a single host.  So when I run 4 test Nmaps, that brings the total  Nmap
> count up to 20.
>
> > That's the best explanation I can think of. Otherwise, your 2048-2048
> > result doesn't jibe with the one from a few days ago:
> >
> > > Nmap done: 186368 IP addresses (15804 hosts up) scanned in 258.023
> > > seconds Raw packets sent: 1398404 (55.937MB) | Rcvd: 51659 (2.426MB)
>
> Okay, I ran the NOMIN-NOMAX and 2048-2048 scan, one after another with the
> box otherwise completely idle:
>
> 2048-2048: (david_new_4c.svg)
> # Nmap done at Thu Sep  6 20:44:41 2007 -- 186368 IP addresses (11113 hosts
> up) scanned in 942.198 seconds
>
> NOMIN-NOMAX: (david_new_4d.svg)
> # Nmap done at Thu Sep  6 17:13:47 2007 -- 186368 IP addresses (12902 hosts
> up) scanned in 2726.080 seconds

I don't understand why the results should diverge so much, especially as
you've explained it shouldn't matter with your machine's connection.
> From your last round of testing:

> 2048-2048: david_new_2c.svg)
> # Nmap done at Thu Sep  6 01:16:10 2007 -- 186368 IP addresses (7656 hosts
> up) scanned in 4066.421 seconds
>
> NOMIN-NOMAX (just -T5): david_new_2d.svg)
> # Nmap done at Thu Sep  6 01:47:37 2007 -- 186368 IP addresses (8972 hosts
> up) scanned in 5943.742 seconds

> > The only other possibility I can think of is that the additional drop
> > processing is slowing it down. I had previously ignored dropped ICMP
> > destination unreachables; those are no longer ignored because I thought
> > the scaled congestion control would more than make up for the drops. If
> > the above tests don't give favorable results, please try again with the
> > attached patch (it's just the reverse of r5780). Ignoring those drops
> > was something of a compromise to begin with. I think not ignoring them
> > is more correct.
>
> Okay I applied this patch.  There was a 31 line offset against the current
> SVN so I had to specify a higher fuzz factor.  I think ignoring the
> drops (destination net and destination host) is probably the right way to
> go.  I'm interested in hearing your reasoning for why they shouldn't be
> ignored.
>
> 2048-2048: (david_new_5c.svg)
> # Nmap done at Thu Sep  6 18:21:49 2007 -- 186368 IP addresses (11217 hosts
> up) scanned in 941.418 seconds
>
> NOMIN-NOMAX: (david_new_5d.svg)
> # Nmap done at Thu Sep  6 19:07:28 2007 -- 186368 IP addresses (13074 hosts
> up) scanned in 2721.055 seconds

We don't treat every destination unreachable as a drop, only those that
are responses to a retransmitted probe. Presumably the first probe or
its ICMP response was dropped. In any case, your tests show almost
identical times either way.

> What SVG viewer do you use?  Everything I've tried has been rather unwieldy.

I use rsvg-view for previews and Inkscape for editing and exporting.

> Although the standard '.nmap' output of one of these scans is several
> hundred megs, I can make them available to you (privately) if you think it
> will help you track down what is going on better that your graphs will.

I'll let you know about that. I'm curious about the long stretches in
your latest "d" graphs--these must be blocks of 4096 addresses with no
hosts up in them. Do you get any "(0 hosts up)" in the log files? Those
would certainly slow the scan down, because congestion variables are not
shared between host groups.

I don't know what to do right now. Maybe I'll catch some more
inspiration on the way to school. I hate to do this, but could you run
those two scans again with r5778 from nmap-massping-migration? Maybe it
will help me understand why these latest changes have such a bad effect
on your tests, especially when they make mine so much better.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org