Re: New development in host discovery: response rate scaled congestion control

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Well I've done it again.  The reason these test don't make sense is because
I screwed them up.  My template was pointing at the wrong binary.  I hope I
haven't caused too much undue stress or panic.  In the middle of re-doing
with r5778 I realized my template was messed up.

I shouldn't be mixing production and development scripts on the same
machine but we don't have our scanning dev box up yet.

I'm repeating these tests with various versions of the
nmap-massping-migration branch and should have results this evening or
tomorrow.

Once again, I apologize for reporting inaccurate information.

Brandon


On Thu, 6 Sep 2007 16:04:04 -0600
David Fifield <david () bamsoftware com> wrote:

> On Thu, Sep 06, 2007 at 08:53:33PM +0000, Brandon Enright wrote:
> > David Fifield <david () bamsoftware com> wrote:
> > > You said that you ran all the scans at the same time. This new
> > > technique ought to approximate TCP congestion control more
> > > closely--if four concurrent Nmaps are run, each one should run about
> > > 1/4 as fast.
> >
> > This makes sense theoretically.  Practically though, this machine is on
> > gigabit Ethernet and capable of sending hundreds of thousands of small
> > packets a second.  I think the limiting factor in sending packets is the
> > module for the NIC, OS buffers, and syscall servicing rate.
> >
> > This machine normally has a minimum of 16 nice'd Nmap running, each
> > against a single host.  So when I run 4 test Nmaps, that brings the
> > total  Nmap count up to 20.
> >
> > > That's the best explanation I can think of. Otherwise, your 2048-2048
> > > result doesn't jibe with the one from a few days ago:
> > >
> > > > Nmap done: 186368 IP addresses (15804 hosts up) scanned in 258.023
> > > > seconds Raw packets sent: 1398404 (55.937MB) | Rcvd: 51659 (2.426MB)
> >
> > Okay, I ran the NOMIN-NOMAX and 2048-2048 scan, one after another with
> > the box otherwise completely idle:
> >
> > 2048-2048: (david_new_4c.svg)
> > # Nmap done at Thu Sep  6 20:44:41 2007 -- 186368 IP addresses (11113
> > hosts up) scanned in 942.198 seconds
> >
> > NOMIN-NOMAX: (david_new_4d.svg)
> > # Nmap done at Thu Sep  6 17:13:47 2007 -- 186368 IP addresses (12902
> > hosts up) scanned in 2726.080 seconds
>
> I don't understand why the results should diverge so much, especially as
> you've explained it shouldn't matter with your machine's connection.
> > From your last round of testing:
>
> > 2048-2048: david_new_2c.svg)
> > # Nmap done at Thu Sep  6 01:16:10 2007 -- 186368 IP addresses (7656
> > hosts up) scanned in 4066.421 seconds
> >
> > NOMIN-NOMAX (just -T5): david_new_2d.svg)
> > # Nmap done at Thu Sep  6 01:47:37 2007 -- 186368 IP addresses (8972
> > hosts up) scanned in 5943.742 seconds
>
> > > The only other possibility I can think of is that the additional drop
> > > processing is slowing it down. I had previously ignored dropped ICMP
> > > destination unreachables; those are no longer ignored because I
> > > thought the scaled congestion control would more than make up for the
> > > drops. If the above tests don't give favorable results, please try
> > > again with the attached patch (it's just the reverse of r5780).
> > > Ignoring those drops was something of a compromise to begin with. I
> > > think not ignoring them is more correct.
> >
> > Okay I applied this patch.  There was a 31 line offset against the
> > current SVN so I had to specify a higher fuzz factor.  I think ignoring
> > the drops (destination net and destination host) is probably the right
> > way to go.  I'm interested in hearing your reasoning for why they
> > shouldn't be ignored.
> >
> > 2048-2048: (david_new_5c.svg)
> > # Nmap done at Thu Sep  6 18:21:49 2007 -- 186368 IP addresses (11217
> > hosts up) scanned in 941.418 seconds
> >
> > NOMIN-NOMAX: (david_new_5d.svg)
> > # Nmap done at Thu Sep  6 19:07:28 2007 -- 186368 IP addresses (13074
> > hosts up) scanned in 2721.055 seconds
>
> We don't treat every destination unreachable as a drop, only those that
> are responses to a retransmitted probe. Presumably the first probe or
> its ICMP response was dropped. In any case, your tests show almost
> identical times either way.
>
> > What SVG viewer do you use?  Everything I've tried has been rather
> > unwieldy.
>
> I use rsvg-view for previews and Inkscape for editing and exporting.
>
> > Although the standard '.nmap' output of one of these scans is several
> > hundred megs, I can make them available to you (privately) if you think
> > it will help you track down what is going on better that your graphs
> > will.
>
> I'll let you know about that. I'm curious about the long stretches in
> your latest "d" graphs--these must be blocks of 4096 addresses with no
> hosts up in them. Do you get any "(0 hosts up)" in the log files? Those
> would certainly slow the scan down, because congestion variables are not
> shared between host groups.
>
> I don't know what to do right now. Maybe I'll catch some more
> inspiration on the way to school. I hate to do this, but could you run
> those two scans again with r5778 from nmap-massping-migration? Maybe it
> will help me understand why these latest changes have such a bad effect
> on your tests, especially when they make mine so much better.
>
> David Fifield
>
> _______________________________________________
> Sent through the nmap-dev mailing list
> http://cgi.insecure.org/mailman/listinfo/nmap-dev
> Archived at http://SecLists.Org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFG4JbhqaGPzAsl94IRAmh+AJsGBgpRCULb+9kgNcapg1AFQvYuNgCgv1DF
dJFpeT4Xblg+cf8K+uBSXZA=
=65Kz
-----END PGP SIGNATURE-----

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org