nmap massive memory usage

[John Richard Moser <nigelenki () comcast net>]

In scanning two /24 ranges for full UDP sweeps on all 0-65535 ports, I 
found nmap (after fininshing the first group) held 600MB+ of RSS on 
64-bit Linux.  Command line:

nmap -oA nmap.udp -vv -sU --min-hostgroup 64 --min-parallelism 128 \
  -T4 -p0-65535 -P0 192.168.1.0/24 192.168.2.0/24

I HAVE NOT READ THE CODE.  I am guessing blindly at how nmap works and 
making notes.  This message contains long boring sections of "if I were 
to write a tool for this I'd do X Y and Z" so just look for the below 
line when you want to find the next dumb thought.

In case you don't feel like reading my useless rant, core question is: 
why does nmap eat so much memory?  As for the rest of you, I'll probably 
get an explanation of why anything I wrote below does or doesn't work or 
a flame about not sending useless crap ;)

== BREAK SECTION ==


I calculate if nmap uses 8192 bytes of storage per bitmap to create 6 
bitmaps per host (one for each open, closed, filtered, unfiltered, 
open|filtered, closed|filtered), it will eat 3MiB per 64 hosts (so in 
this case, up to 24MB) just for that.  That should get flushed each time 
a host's information gets flushed to disk, so not a problem.  3MiB 
accounted for.

Note:  You can use 3 bit to track the state here, and have nmap use 
24576 bytes instead of 49152 per host.  To look up the bits, nmap would 
have to have an array of char[24576] as a bitmap and find the starting 
index by.  This would even give a state for "Unscanned" to help track 
randomized ports (and hosts).

===

/* we write LTR so offsets...
  00000000
  01234567

  Bits

  00000000 00000000
  01234567 89111111
             012345 */
bit    = (port * 3);
byte   = bit / 8;
offset = bit % 8;
val = bitmap[byte];
if (offset > 5) {
        /* shift over to make room
         00000011 10000000
                <-
         00000110 10000000 */
        val <<= (3 - (8 - offset));
        /* OR with MSBs of next byte shifted to the right
         00000110 10000000
                   ------>
         00000110 00000001
         (bit OR)
         00000111 */
         val |= (bitmap[byte+1] >> 8 - (3 - (8 - offset));
}
else {
        val >>= (8 - offset) - 3;
}
/* clear to the 3 bits we need */
val &= 0x07;

===

nmap could also just use a linked list of ports, but that would quickly 
outgrow this:

32-Bit  64-bit  Desc
4       8       Pointer to next
2       2       Short int:  port
1       1       Byte: state

The compiler would likely pad this to 8 bytes and 12 bytes, 
respectively.  So after 6144 or 4096 ports (for 6 individual bitmaps), 
or 3072 or 2048 ports (for the above compacted single not-bitmap), 
memory usage would pass the bitmap.

Using an array instead, and a separate list of ordered ports (states[5] 
versus {21,22,80,443,3306} as a port list), it's still just 1 byte each, 
which is better until you reach 24576 ports (and thus not important 
enough for a complex hack).

Of course, service version scanning makes this moot; the problem is no 
longer simple, and the array carries tons of extra data.

===================
== BREAK SECTION ==
===================

So what else could hold memory?  Obviously the above will eat at most 
maybe 24MB in my entire scan.  What gets stored?  Packet captures? 
Interesting information about the system?  Does any of that get flushed 
once the host is scanned?

Once a host is done being scanned, it should immediately be analyzed and 
then have any relevant data not useful for further analysis freed.  I 
can't imagine that nmap is holding anything else in there though, why 
would it hold packet captures and all beyond useful life?


-- 
Bring back the Firefox plushy!
http://digg.com/linux_unix/Is_the_Firefox_plush_gone_for_good
https://bugzilla.mozilla.org/show_bug.cgi?id=322367

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org