Re: Replacing passwords.lst

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> The sizes were not as bad as I thought at first. After stripping extra
> spaces, we are left with
>
> -rw-r--r--  1 david users  88K 2010-03-16 17:13 faithwriters.lst
> -rw-r--r--  1 david users 103K 2010-03-16 17:14 hotmail.lst
> -rw-r--r--  1 david users 421K 2010-03-16 17:07 myspace.lst
> -rw-r--r--  1 david users 1.9M 2010-03-16 17:18 phpbb.lst
> -rw-r--r--  1 david users  58M 2010-03-16 17:24 rockyou.lst.bz2
>
> I wrote a simple program to sum the counts from several password files
> and output the top n passwords. Using the five lists above, I
> regenerated our nselib/data/passwords.lst. The program automatically
> does bz2 decompression based on filename so keeping compressed lists
> isn't inconvenient.

Cool, it's good to handle the bz2 compression transparently.  I think
we can't just sum the lists though without normalizing them to a
degree.  Otherwise rockyou is weighted too strongly.

Ron and I chatted off-list about this a bit.  A simple linear weight
probably isn't the right choice because things that are only duplicated
a few times in phpbb or mypspace would get scaled up too much.

I think if we plot the most common N passwords for N in the 5k-10k
range we'll probably get a exponential curve with a long tail.
Whatever the shape of the curve, once we have a model for it, we should
normalize the area under each list's curve for the first N passwords
and just assign them a floating point value of rank for each list.  We
can sum the ranks and order them.

This is something I have been planning on doing but haven't had time to
do yet.

Brandon

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iEYEARECAAYFAkugJuEACgkQqaGPzAsl94KUvgCgrWPFQ0Z6IDQdiTd4r5lGb2J/
gJoAnAlRdaahm+nziaS3SYKhmvudp6hA
=1h5W
-----END PGP SIGNATURE-----
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/