Re: Replacing passwords.lst

[Fyodor <fyodor () insecure org>]

On Fri, Mar 12, 2010 at 09:13:09PM -0700, David Fifield wrote:
> I made this directory and copied the old MySpace passwords into it. I
> didn't realize that Ron's databases were so huge--RockYou is like 100
> MB. I copied the first 10,000 lines of phpBB and RockYou into the
> directory as well.

Yeah, that is huge.  And I know I sometimes complain about stuffing
large files in SVN.  But this is really useful data, so I'd support
storing more.  Rockyou is the biggest issue, as you noted.  My
suggestion for that is:

o Only about 2.5 million of the 14 million rockyou passwords are seen
  more than once.  So we can dramatically reduce the file size by
  limiting it to passwords seen at least twice.  The unique ones aren't
  as valuable to us anyway.

o We can reduce the file size a bit more by removing column alignment
  spaces.  We're going to be parsing these with applications so I'd
  remove any leading spaces and all but one trailing space from the
  counts.  I think the passwords may be allowed to contain space
  chars, so using "awk '{print $1,$2}'" may cause data loss.

o We should probably bzip2 any of the password files which are more
  than a few megabytes.  That makes them smaller in the repository,
  and we can always uncompress them when we're actually using them.

o Most (maybe all) of the other password files are probably small
  enough that we can skip the removal of unique passwords.

If these were different sorts of files, I'd say just include a note
with the URLs for them.  But password files tend to go away, so it is
worth saving our own copies in SVN IMHO.

> Are there any others that are recommended as
> general-purpose lists?

Good question.  I hope folks will speak up if you have ideas!  And
thanks to Ron for collecting all this.

Cheers,
-F
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/