Tudor's Status Report - #16 of 17

[Tudor-Emil COMAN <tudor_emil.coman () cti pub ro>]

Hi,



I followed Dan's advice for determining the most used ports by scanning batches of 20K hosts on 4 ports each.

The exact scan I did was:


#!/bin/bash


LOW=1
HIGH=4

for (( i=0; i<16384; i=i+1))
do
  shuf -n 20000 nmap-ultra/hostlist > current_hostlist
 ./nmap/nmap -iL current_hostlist --min-rate 140000 --min-hostgroup 10000  -n -Pn  --max-retries 0 &> /dev/null -oX 
ports${i}b.xml -sS -p ${LOW}-${HIGH} -T5
  LOW=$(($LOW+4))
  HIGH=$(($HIGH+4))
done


It generated 199GB of XML logs and currently they are being processed by the nmap-top-ports.py script.
I'll do a follow-up mail when that's done with the differences between the current top ports file and my findings.


While testing on Windows I experienced some crashes that I traced back to my changes regarding avoiding calls to 
target_needs_hostgroup.
I don't really know why that happened but I reverted the code that I added to avoid those calls.
That change would only scale with the amount of targets in a hostgroup and not the number of ports that were scanned, 
so it was about only a couple of seconds per scan.


I spent some time working on IOCP and doing some changes based on Henri's advice.
Adding the event initiation/termination in iod_register/iod_modify seems to be causing some problems and I'm still 
working on that.
I disabled some internal structure reuse so it's a bit slower than it should be while I figure out what's wrong.
I tested if pcap events work in engine_iocp.c by using nping and it seems to be working fine.
I had some concerns about memory usage but I made only pure read events (that doesn't include pcap_read events) carry 
that buffer.
I tried making that read buffer smaller to see if it affects performance but I couldn't see any visible increase.
Localhost scanning works with -sT in case anybody was still wondering about that.


Accomplishments:
- Did a findHost() implementation benchmark: logarithmic search vs linear search.
- Changed the Nsock API to integrate IOCP better.
- Completed the top-ports scan.
- Did some new IOCP benchmarks but with some improvements disabled so it is a bit worse that usual but still beats poll.

Priorities:
- Complete the IOCP engine.
- Finish analyzing the logs generated from the top-ports scan.
- Create a performance guide.

Cheers,
Tudor

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/