SoC Feature Creeper and Performance Czar Tasks

[Fyodor <fyodor () insecure org>]

While some of the Google-sponsored students will be working on a
single large project for the whole summer (like a new GUI, or adding
scripting support to Nmap), another group called the "feature
creepers" will each have a number of smaller tasks which will allow
them to explore the whole code base and contribute in many different
areas.  There will also be (probably) one person who focuses on
performance/QA and code cleanup tasks.  I have created the list below
of some tasks I would like to see see done this summer.  They vary
from simple tasks that shouldn't take more than an hour (like -q), to
much larger projects that could take a week or two (like fixed-rate
packet sending, traceroute, or application scanning).  I'll do as many
as I can myself, and I'm sure the feature creepers will finish a bunch
of them.  Also, if any of you get the itch to do a little Nmap
development work, this is a great list to pick ideas from.  Code
fixes/improvements are accepted from all quarters!  It is best if you
let me know when you are starting work so I can make sure someone else
isn't doing the same thing.  Also, if you have a pet peeve about Nmap
or an improvement idea, post it to nmap-dev as a possible addition to
the list.

With all that, here is the initial task list:

[ GENERAL TASKS ]

o Fix Nmap so that regardless of order given, -T arguments are
  processed before specialized timing args.  It may be easiest to
  store the specialized arguments in variables until option processing
  is done, and then process them.  Or set flags such as max_rtt_set
  when you set that, then the -T option processing could check the
  flag before mucking with the rtt.  Currently, you need to specify
  the -T option before any specific timing options like
  --max-rtt-timeout.

o Add parallel traceroute support to Nmap.  It should come after the
  port scanning and host enumeration sections, and utilize a
  port/protocol that Nmap has found to be accessible on the particular
  target.  Multiple hosts should be done in parallel (as with most
  Nmap scans), and multiple probes done in parallel too.  The Nmap
  parallel rDNS infrastructure should be used to look up the
  intermediate hosts (unless -n was specified), because that system
  does caching (be sure to verify that they are indeed being cached,
  as many of the intermediate hosts will be repeated for each
  machine).  Printing to normal output should be concise -- for
  example you can show the whole path for the first system and then
  only show for the first point of path divergience for the next
  system.  You may even be able to do the probes backwards and cease
  probing once you find a machine that you already know the path to.
  XML output format should give the entire route for each host, since
  space isn't as serious of a constraint there.

o Fix UDP scan such that it doesn't find its own port open when
  scanning localhost (try nmap -sU -p- localhost).

o Remove inet_aton.c from nbase and change all Nmap calls to
  inet_pton.  The duplication is pointless, plus the present
  inet_aton.c is BSD licensed WITH ADVERTISING CLAUSE.

o Remove massping() in favor of doing all host enumeration from
  ultra_scan().
  o Once this is done, make default host host discovery more
  comprehensive, at least when !isdirectlyconnected().

o Maybe should add an option like --reason which adds a column to the
  port table explaining why each port is considered up/down/whatever.
  A typical reason might be "no response" or "port-unreach from
  XX.XX.XX.XX"

o Write a general scanning engine for abusing applications for port
  scanning purposes. This would handle scanning through SOCKS and HTTP
  proxies, and the existing FTP bounce scan would also be ported to this
  engine. Proxy chaining must be supported.

o Explore many different application protocols, trying to find ways
  for version detection to identify difficult services such as the new
  Skype protocols or DHCP.

o Solar Designer notes:
  "When there're ports in more than two categories (say, open, closed,
   and filtered), the output is often way too long.  For example, it
   is rather common to filter out ports below 1024 except those
   actually used for public services.  Currently, Nmap would report
   the fact that "ports not listed are filtered", followed by a long
   list of all the "closed" ports it scanned with some "open" ones
   inbetween.  Unless I grep out the "closed", it's very inconvenient
   to see which ports are open.  And I often don't know a scan will
   produce this kind of result before I start it, so I don't redirect
   Nmap output anywhere or pipe it through grep, wait for a long time
   for it to complete the scan, and then have my terminal flooded..."
     o Solution:  If there are more than, say 50 ports in a non-open
       state other than the ignored state, simply list them (comma
       separated, with ranges, like the XML output does with port lists).

o Add -q (quiet) mode, which only prints open ports (maybe just a line
  specifying how many closed/filtered ports there are for a host, if
  there are any).  This option may also omit certain low-priority
  messages.  This would replace the undocumented "quash argv"
  functionality of -q.  The code for that should accordingly be
  removed (if anyone uses that quash argv feature -- speak up now!).

[ PERFORMANCE/QA TASKS ]

General Performance Notes
 o While performance is very important, be wary of any performance
   improvements that:
   o Are likely to reduce Nmap accuracy in some realistic cases
   o Make the code/APIs significantly uglier, harder to use, easier to
     use insecurely, or otherwise disadventageous.
   o Is not portable
 o Performance improvements should generally be based on benchmarks
   showing that there is a problem in the first place, and demonstrating
   that the fix helps.

o [PERF] Run valgrind and other memory debugging stuff. I will
  discover memory leaks and calls to malloc that can be optimized.  Add
  a command-line option which has Nmap free as much memory as practical
  before quitting, to aid detection of true memory leaks.

o [PREF] Run Windows performance/quality testing tools (maybe through
  trial versions where they are commercial).  Benchmark Windows
  vs. Linux Performance and look at ways to improve areas where Windows
  is much slower.  Poll nmap-dev for ideas as well.
  o There are some apparently good suggestions for improving Nmap
    compilation on windows (simple things, like changing optimization
    flags) in 7/16/03 mail from "bingle" <bingle2000(a)hotmail.com>
  o Windows TCP connect() scan may be limiting itself to too few
    sockets, because it looks like max_sd() always returns zero for
    Windows.  Investigate.  Maybe I should limit it to 9 due to the new
    SP2 limitations.

o [PERF] Scan a large number of machines across the Internet, look at
  accuracy and timing issues. Try to improve algorithms without
  sacrificing accuracy. Do similar tests on a LAN. Since Nmap can
  detect whether a host is on a LAN or has to go through a router,
  maybe Nmap should use more aggressive default options in the former
  case. And even more aggressive if Nmap detects that it is scanning
  localhost.  Nmap has functions like islocalhost() and
  isdirectlyconnected() to easily detect these conditions.  Tune the
  various Nmap performance values, such as those in
  init_perf_values().  Be very, very careful not to sacrifice accuracy
  for performance.

o [PERF] Run profilers (such as gprof) against Nmap, identify and fix
  the resource hogs

o [PERF] Add a fixed-rate packet sending mode to ultra_scan() (like
  what UnicornScan and ScanRand offer).  Must be done cleanly since
  this is critical Nmap engine code.

o [PERF] dentify and eliminate dead and duplicate code.There are
  probably many functions in the Nmap code base which aren't even
  used. There are also functionality duplication (like having tcpip.h
  packet headers and the same in dnet) which could be removed by
  standardizing on one or the other.

o [PERF] Rewrite Portlist class in a more efficient way, which takes
  advantage of the way it is used (e.g. out-of-order inserts are very
  common, searches are very common, sorted iteration is common). It
  uses too much CPU time right now.

o [PERF] Investigate ICMP error message handling -- see (
  http://xtrmntr.org/ORBman/tmp/nmap/ ) for an example patch, also see
  http://seclists.org/lists/nmap-dev/2006/Apr-Jun/0054.html

o [PERF] Go through tcpip.h and such and remove extraneous stuff,
  standardize on dnet packet headers, etc.

o [PERF] I need to fix pcap so that it has a proper timeout, which is set for
  each pcap_next.  Currently, I just set a very low timeout in open_live
  and then my readip_pcap will call it over and over until the real
  timeout is reached.
  o Consider changing Nmap to achieve non-blocking pcap read by
    setting the pcap sd nonblocking on Linux rather than my patch.
    See 1/16/03 mail from Pavel Kankovsky to Nessus list. [ Adding a
    general timeout to pcap may be better. ]

o [PERF] Determine empirically which TCP and UDP ports are most
  commonly open.  Add an easy way to scan just the most popular ports.
  Maybe this could be an extra designation in nmap-services, or maybe
  ordered arrays in the code like top_tcp_ports[] and top_udp_ports.
  Maybe -F should scan (just) all of the popular ports, and perhaps
  there should be an option to specify how many popular ports you want
  for each scanned protocol (e.g. --top-ports 24).


Cheers,
Fyodor


_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev