Re: Updates to http-enum.nse

[Fyodor <fyodor () insecure org>]

On Thu, Aug 20, 2009 at 11:57:43AM -0500, Ron wrote:
> Me and one of my minions at work (Andrew -- same guy who I did the iis 
> unicode script with) have put a lot of work into improving http-enum.nse 
> (in case that wasn't obvious from all the http.lua errors I've been 
> posting). Rob's script was a great start, but we made a ton of improvements:
> - Cleaned up the code, put a bunch of it into functions
> - Support for many more HTTP status codes
> - Improved detection for 404 pages (especially those that return 200) -- 
> we still have some more work to do on this, but it's getting there
> - More intelligent usage of HEAD vs. GET requests
> - Ability to parse external fingerprint file (attached)

Thanks Ron and Andrew!  Those sound like exciting changes!  It worked
well in my testing, though the results against scanme.nmap.org are
basically false positives (we might wan to consider only showing 200
results by default--I'm not sure).

The format of the fingerprint file is a bit questionable.  Comments
lines starting with '#' are parsed and then printed in the script
output when paths given later in the file are discovered.  I realize
you didn't invent this format, but it is so simple that it could
easily be improved.  For example, each path could include the
description on the same line.  Or there could be a keyword introducing
the description on one line, followed by the paths on their own lines
as they are now.  Then we could use comments for notes which we don't
want parsed and printed by http-enum.  In deciding on the format, it
may be worth thinking about how it could be extended if we want to
include more information later (just as an idea off the top of my
head, we might want to later indicate what status code we look for to
address issues such as http-enum reporting that scanme.nmap.org has
"TeraStation PRO RAID 0/1/5 Network Attached Storage" just because
/cgi-bin/image/shikaku2.png shows forbidden).

Another serious issue involves inclusion of the Yokoso DB.  You say:

> That last point is the interesting one, to me -- we use the same file 
> format as the Yokoso project (by Kevin Johnson and others, from Intel 
> Guardians). This lets us leverage their fingerprints as well (and 
> they've given me permission to include a copy of their fingerprints 
> file, too, 

That was nice of them, but it is important to get more clarification
and more explicit permission whenever we include 3rd party code or
data into Nmap.  I hate dealing with copyright stuff as much as the
next guy, but we really need to be very careful about this sort of
thing.  When they say we can include the DB with Nmap, what does that
really mean.  Remember that Nmap is open source, so people can
incorporate parts into other projects or fork Nmap under a different
name.  A strict reading of "you may include this file within Nmap"
would not allow such things, which would mean that part of Nmap is not
open source.  Also, a strict reading might mean that we can only
include the file and not modify it (create derivative works).  In
general, we can put third party code/data in with Nmap if it is given
to us under one of the following licenses (either via special
permission or because the code is already under such a license):

o Public domain -- that means people can do whatever they want with it.

o BSD-style (includes MIT license, Lua license, etc.) - preferably
  2-clause variant.  If it has the advertising clause, we need to
  mention it explicitly in the man page
  (http://nmap.org/book/man-legal.html#third-party-soft) and
  potentially other places.

o Nmap license - if they're OK with us distributing it under the terms
  of the Nmap license (http://nmap.org/data/COPYING), that is OK too.

So if they let us use the data under one of these licenses, inclusion
with Nmap is OK.  In any case, the license permission granted needs to
be included (described) at the top of the file.  We only need license
rights to the list of URL paths and descriptions, and not the rest of
Yokoso.

Note that even when a data file isn't licensed appropriately for
distribution with Nmap, we can generally point to it in the
documentation (e.g. if it is a URL somewhere) so users can download
and use it.

> For those who were there (and I know several of you were, because I
> was sitting with you :) ), there was a presentation about Yokoso at
> Defcon.

Haha, yes that was a good talk :).

> One thought I had -- http-enum.nse and Yokoso sort of have different 
> points. http-enum.nse is designed for finding common locations, like 
> /icons, /scripts, /test, etc, and Yokoso is designed for fingerprinting 
> common web apps. So, for that reason, it might make sense to put it in a 
> different script that the user can run separately. Or maybe not. I'm 
> happy with going either way.

I'm not sure, but my gut reaction is that with a good file format
(which doesn't have to be too complex), we could probably combine the
Yokoso DB with the existing enum DB.  There is also a DB by HD Moore
(a NASL script he wrote) which I hope to request permission for us to
use.

If we end up with more URLs than we want to scan by default, we could
look at either splitting it up into multiple scripts or introducing
NSE arguments to select categories of paths to try.

> I plan to move the hardcoded tests from http-enum.nse into their own 
> file, too, once I'm happy with how it's working.

Or maybe one combined file.

Thanks again for all your efforts on this!

Cheers,
-F


_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org