Re: Tudor's Status Report - #13 of 17

[Tudor-Emil COMAN <tudor_emil.coman () cti pub ro>]

Dan,


Well it's like you said, that check is already done for batches of 4096 targets in targets.cc::refresh_hostbatch().

You only need to call target_needs_new_hostgroup in nmap.cc if you are combining targets from different batches.


Let's say you are scanning 5000 hosts, the first 1000 are down, the rest are up. You specified --min-hostgroup 5000.

> From the first batch you get 3096 Targets in your hostgroup( the first 1000 are down). Only for the next 904 from the 
> second batch you actually need to do the check for that hostgroup.


For a -Pn scan(all hosts are considered up), having the o.ping_group_sz match the size of the hostgroup means that 
calling target_needs_new_hostgroup in nmap.cc is redundant.

I'm currently trying to see if modifying that value is safe.


The part from target_needs_new_hostgroup that's slow is iterating through all the targets currently in the hostgroup 
and check if one of them has the same IP address with the one you are  trying to add, it seems to make a lot of 
difference for large scans.


Increasing the size of the host discovery hostgroup to match a potential bigger hostgroup value given by the user 
should make the scan faster anyway.


Thanks,

Tudor

________________________________
From: Daniel Miller <bonsaiviking () gmail com>
Sent: Tuesday, July 26, 2016 4:41:11 PM
To: Tudor-Emil COMAN
Cc: dev () nmap org
Subject: Re: Tudor's Status Report - #13 of 17

Tudor,

I have some questions regarding these performance improvements.

- Removed some unnecessary calls to target_needs_new_hostgroup() in nmap.cc. This one was really a performance killer 
and I made so that for hostgroups smaller than PING_GROUP_SZ (currently 4096) it doesn't get called at all.

Are we sure this can be taken out? I see two sides to this: first, the call is important because if a target gets into 
the wrong hostgroup, it could result in sending packets out the wrong interface or with the wrong source address, etc. 
The other side of it is that this sorting should have already been done in the host discovery ("ping scanning") phase 
by the function of the same name in targets.cc. I guess the code might be easier to understand directly, but could you 
explain a bit more why this improvement is possible?

Performance gains are visible for really high packet rates.

For something like: ./nmap  54.239.156.68/16<http://54.239.156.68/16> -sS -Pn -p 80 -T5 -n --open --min-rate 130000 
--min-hostgroup 16384

Without these optimizations the scan took about 9.6 seconds, with these optimizations it takes about 8.5 seconds.

Are you also checking smaller scans (CIDR /24 is typical) and normal packet rates to be sure that we're not regressing 
in more typical cases?


Thanks for your hard work!
Dan

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/