RE: Password profiling in NSE

[Giacomo Mantani <giacomo.mantani () studio unibo it>]

Hi George and all,

I am a GSoC student interested, it seems useful. I could help you develop some
features if you want and the community approve your work.  Independently from be
approved or not in the summer of code program (results will be public on 25
April).

I have worked in an HTML parser library in order for the GSoC proposal, it
enhance http-title and other script, maybe we could use it to do a better
information gathering phase.

I have never used before cupp[1], mutator[2] nor cruch[3] but I will dig them in
the next days.

Speaking about more complicated mangling algorithms, password often are made up
only with characters that are in the keyboard. I think that really few people use
characters outside (not displayed). We could leverage this supposition to
get a subset of character to use in mutation algorithm. In addition, we could
take advantage of the target country to guessing the keyboard layout.


Have you already a repository with some version control software?


Let me know how I could contribute,

Cheers,
Giacomo Mantani

[1]: https://github.com/Mebus/cupp
[2]: http://www.toolswatch.org/2013/10/mutator-v0-2-wordlist-mutator-released/
[3]: https://sourceforge.net/projects/crunch-wordlist/

________________________________________
From: dev [dev-bounces () nmap org] on behalf of George Chatzisofroniou [sophron () latthi com]
Sent: Friday, April 15, 2016 6:25 AM
To: nmap list
Subject: Password profiling in NSE

Hi guys,

I'm attaching a patch that introduces password profiling to NSE.

System administrators and users tend to show the same patterns when it
comes to passwords; many times they contain the name of the product
they are selling or the company's name or the role of the server in
their infrastructure. Character substitutions with leet, random
capitalization or appending some "random" suffixes are methods that
are far from secure but widely used.

When I'm working as a pentester, during the reconnaissance phase, I
usually gather any keywords that could work as passwords and then pass
those to tools like CUPP [1], Mutator [2] or Crunch [3] to create a
word-list specifically for the organization. I later import the
generated word-list when I'm brute-forcing a service on the run-time
(i.e. with --script-args brute.file in NSE).

NSE can automate the above process in a clever way as it contains
plenty of scripts that perform information gathering as well as others
that perform brute forcing for various protocols in the application
layer. I find this a big advantage of NSE over other brute forcing
tools.

After you apply my patch, you can trigger the new feature with the
"--script-args brute.passprofile=1" option. Here's an example:

./nmap -n -Pn -p21,80 -oN output-local-pwdprof --script
http-title,ftp-brute --script-args
brute.passprofile=1,unpwdb.timelimit=0 -d 127.0.0.1

Nmap scan report for 127.0.0.1
Host is up, received user-set (0.000026s latency).
Scanned at 2016-04-14 15:38:36 CDT for 1312s
PORT   STATE SERVICE REASON
21/tcp open  ftp     syn-ack ttl 64
| ftp-brute:
|   Accounts:
|     admin:NM4p! - Valid credentials
|_  Statistics: Performed 7273 guesses in 1312 seconds, average tps: 3
80/tcp open  http    syn-ack ttl 64
|_http-title: Nmap

We ran two scripts: http-title and ftp-brute. Http-title ran first,
retrieved the HTML title (in this case "Nmap") and then since I had
instructed Nmap to use the password profiling feature, the pwdprofile
library generated a list of mutations based on the "Nmap" keyword that
the brute force engine eventually loaded and used against the FTP
service. One of them ('NM4p!') appeared to be valid for the 'admin'
account.

To implement the feature I created a new NSE library: pwdprofile. It
provides a method for the NSE scripts to pass any keywords to the
library (using NSE registry), mangling algorithms, as well as methods
for adding the generated wordlist to the passwords iterator used by
the brute force engine.

The way it works is rather simple: NSE scripts that obtain certain
keywords that could be possible passwords use the NSE registry to
cache them. Brute.lua library will check if the brute.passprofile
argument is set. If it is, it will call unpwdb.add_profiled_passwords
method that will in turn call pwdprofile.get_profiled_pwds method that
will eventually retrieve all the cached keywords, peform some mangling
and return them as a Lua table. Pwdprofile.lua will merge this table
with the one it retrieves from the text file (usually passwords.lst)
and return an iterator back to brute.lua. Brute scripts, that are
instructed to depend on the discovery scripts, will now load an engine
that iterates first on the passwords retrieved from passwords.lst (or
another text file) and then the generated wordlist from pwdprofile.

An NSE developer that writes a discovery script and believes that some
of the gathered information might be useful for password profiling,
can simply use pwdprofile.save_for_pwdprofiling(host, keyword) method
to pass them to the pwdprofile. This patch implements this for
http-title and smb-enum-domains.

On the other hand, an NSE developer that writes a brute script and
wants his engine to take advantage of this feature when the user sets
the --brute.passprofile argument, only has to add these special
discovery scripts as dependencies. If he is lazy to look for them, he
may use pwdprofile.PWDPROFILE_SCRIPTS table that holds a list with
them. This patch does this for ftp-brute and http-form-brute.

There is of course room for improvement:

* I only leveraged two discovery scripts. There are plenty more that
retrieve possible passwords.

* I introduced the password profiling feature only to a couple of
"brute" scripts. The rest of them can also benefit from it.

* More complicated mangling algorithms. Paulino Colderon had posted a
relevant patch a while back [4]. I also added a few TODO notes here
and there in my code.

* Introduce levels of mangling. Brute forcing can be time consuming.
Sometimes we want to generate a small set of passwords.

* I think NSE lacks of an option that will enforce a script to run
with all of its dependencies. For example, a user may want to say
"Brute force the FTP service after obtaining possible passwords from
whatever other service you can". Right now, the user needs to remember
the list of scripts that the brute script depends on and manually
write them down, and this may be a rather huge list.

* Currently this feature works independently for each host in a
sub-net. I can imagine sub-nets that a keyword retrieved from one host
can work as a password for another.

* There should be an option to skip the local passwords
(passwords.lst) and only use those from the generated word-list.

* In some cases, using pwdprofile.PWDPROFILE_SCRIPTS may result to
circular dependencies. We need a better handling for this.

I can implement the above through time, or even better, a promising
GSoC student can do it (I believe there's a "password wizard" spot).

I already had some discussions about this feature with ithilgore but I
would be happy to get more feedback.

[1]: https://github.com/Mebus/cupp
[2]: http://www.toolswatch.org/2013/10/mutator-v0-2-wordlist-mutator-released/
[3]: https://sourceforge.net/projects/crunch-wordlist/
[4]: http://seclists.org/nmap-dev/2014/q4/382

Cheers,

--
George Chatzisofroniou
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/