3.59ALPHA1: Core port scanner rewrite!

[Fyodor <fyodor () insecure org>]

-----BEGIN PGP SIGNED MESSAGE-----

Hey all,

I spent most of the last month huddled up in my office rewriting the
core Nmap port scanning algorithms from scratch.  The initial results
were so promising, that I skipped Defcon to spend that weekend
coding.  The biggest change is that Nmap can now port- and
version-scan many hosts in parallel.  All of the popular scan types
have been redone to support this, including SYN, Connect, FIN, UDP,
Protocol, ACK, Window, Maimon, NULL, and Xmas scans.  The UDP overhaul
included adding a new state "open|filtered" for ports that do not
respond.  If version scan is used, it will try all of its UDP probes
to elicit any response from these ports.  If it succeeds, the state
changes to opened.  This resolves the false positive issue, where Nmap
would report UDP ports as "open" when they were really filtered.
Another nice new feature is timing stats -- in verbose mode Nmap will
now provide an estimated time of completion when the scan will take
more than a minute or two.  There are many other changes, but I'll let
you read them right from the changelog below.

Unfortunately, such an overhaul guarantees at least a few (or a few
dozen :) bugs and imperfections.  So I'm creating an ALPHA release, in
the hope that some of you will test it out and report any problems
encountered.  A success report once in a while can be nice too :).

I'm particularly interested in tuning and improving the complex timing
system of the new ultra_scan() framework.  So please let me know how
it works for you, particularly if you find a case where Nmap 3.55 is
faster than this new release.

Here are the CHANGELOG entries:

o Rewrote core port scanning engine, which is now named ultra_scan().
  Improved algorithms make this faster (often dramatically so) in
  almost all cases.  Not only is it superior against single hosts,
  ultra_scan() can scan many hosts (sometimes hundreds) in parallel.
  This offers many efficiency/speed advantages.  For example, hosts
  often limit the ICMP port unreachable packets used by UDP scans to
  1/second.  That made those scans extraordinarily slow in previous
  versions of Nmap.  But if you are scanning 100 hosts at once,
  suddenly you can receive 100 responses per second.  Spreading the
  scan amongst hosts is also gentler toward the target hosts.  Nmap
  can still scan many ports at the same time, as well.  If you find
  cases where ultra_scan is slower or less accurate, please send a
  report (including exact command-lines, versions used, and output, if
  possible) to Fyodor.

o Added --max_hostgroup option which specifies the maximum number of
  hosts that Nmap is allowed to scan in parallel.

o Added --min_hostgroup option which specifies the minimum number of
  hosts that Nmap should scan in parallel (there are some exceptions
  where Nmap will still scan smaller groups -- see man page).  Of
  course, Nmap will try to choose efficient values even if you don't
  specify hostgroup restrictions explicitly.

o Rewrote TCP SYN, ACK, Window, and Connect() scans to use
  ultra_scan() framework, rather than the old pos_scan().

o Rewrote FIN, Xmas, NULL, Maimon, UDP, and IP Protocol scans to use
  ultra_scan(), rather than the old super_scan().

o Overhauled UDP scan.  Ports that don't respond are now classified as
  "open|filtered" (open or filtered) rather than "open".  The (somewhat
  rare) ports that actually respond with a UDP packet to the empty
  probe are considered open.  If version detection is requested, it
  will be performed on open|filtered ports.  Any that respond to any of
  the UDP probes will have their status changed to open.  This avoids a
  the false-positive problem where filtered UDP ports appear to be
  open, leading to terrified newbies thinking their machine is
  infected by back orifice.

o Nmap now estimates completion times for almost all port scan types
  (any that use ultra_scan()) as well as service scan (version
  detection).  These are only shown in verbose mode (-v).  On scans
  that take more than a minute or two, you will see occasional updates
  like:
  SYN Stealth Scan Timing: About 30.01% done; ETC: 16:04 (0:01:09 remaining)
  New updates are given when the estimates change significantly.

o Nmap now has a "port scan ping" system.  If it has received at least
  one response from any port on the host, but has not received
  responses lately (usually due to filtering), Nmap will "ping" that
  known-good port occasionally to detect latency, packet drop rate,
  etc.

o The port randomizer now has a bias toward putting
  commonly-accessible ports (80, 22, etc.) near the beginning of the
  list.  Getting a response early helps Nmap calculate response times and
  detect packet loss, so the scan goes faster.

o Host timeout system (--host_timeout) overhauled to support host
  parallelization.  Hosts times are tracked separately, so a host that
  finishes a SYN scan quickly is not penalized for an exceptionally
  slow host being scanned at the same time.

o When Nmap has not received any responses from a host, it can now
  use certain timing values from other hosts from the same scan group.

o Enabled MAC address reporting when using the Windows version
  of Nmap.  Thanks to Andy Lutomirski (luto(a)stanford.edu) for
  writing and sending the patch.

o Upgraded libpcap from version 0.7.2 to 0.8.3.

o Updated nmapfe.desktop file to provide better NmapFE desktop support
  under Fedora Core and other systems.  Thanks to Mephisto
  (mephisto(a)mephisto.ma.cx) for sending the patch.

o Further nmapfe.desktop changes to better fit the freedesktop
  standard.  The patch came from Murphy (m3rf(a)swimmingnoodle.com).

o Fixed capitalization (with a perl script) of many over-capitalized
  vendor names in nmap-mac-prefixes.

o Ensured that MAC address vendor names are always escaped in XML
  output if they contain illegal characters (particularly '&').  Thanks
  to Matthieu Verbert (mve(a)zurich.ibm.com) for the report and a patch.

o Changed xmloutputversion in XML output from 1.0 to 1.01 to note that
  there was a slight change (which was actually the MAC stuff in 3.55).
  Thanks to Lionel CONS (lionel.cons(a)cern.ch) for the suggestion.

o Removed Ident scan (-I).  It was rarely useful, and the
  implementation would have to be rewritten for the new ultra_scan()
  system.  If there is significant demand, perhaps I'll put it back in
  sometime.

o Removed (addport) tags from XML output.  They used to provide open
  ports as they were discovered, but don't work now that the port
  scanners scan many hosts at once.  They did not specify an IP
  address.  Of course the appropriate (port) tags are still printed
  once scanning of a target is complete.

No Windows binaries are provided as I haven't even tried compiling it
on that platform.  If anyone wants to do so and send me a patch, that
would be helpful.  Otherwise, I'll do it sometimes before the
non-ALPHA nmap-hackers release.  Here are the goods:

http://www.insecure.org/nmap/dist/?C=M&O=D

And the checksums:

24fe62ac97aa32794887ff582fc03f81  nmap-3.59ALPHA1-1.i386.rpm
6d8399bdb9674039a411b48c44f5e08f  nmap-3.59ALPHA1-1.src.rpm
eea06145e3d3a100452a327c0927f148  nmap-3.59ALPHA1.tar.bz2
70e234c57ccee8fb3f7e783b4b41e4f4  nmap-3.59ALPHA1.tgz
aac3f918405219d6dd1c5f7c61f9fe00  nmap-frontend-3.59ALPHA1-1.i386.rpm

These release notes should be signed with my PGP key, which is
available at http://www.insecure.org/fyodor_gpgkey.txt .  The key
fingerprint is: 97 2F 93 AB 9C B0 09 80 D9 51 40 6B B9 BC E1 7E

Enjoy!  And please let me know if you find any problems.

Cheers,
Fyodor
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iQCVAwUBQSVGlc4dPqJTWH2VAQHkoQQAly6WnF7TQq2Y+T0+WaEN8NH+08rjx3l9
Ugk2XAJray4w+SI6XG5UGsYeXFqbcdCd3liotnd/jOXpycaf1EzPchM5DUFtHpXN
Gy/x2qTK5zCYbybh6K2wlqZ7N7747brkJjBY6BWzamTyW0zaxPkVru0bguDPrNQM
LbSjWniL3H0=
=djcN
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
For help using this (nmap-dev) mailing list, send a blank email to 
nmap-dev-help () insecure org . List archive: http://seclists.org