Home page logo

nmap-dev logo Nmap Development mailing list archives

some nmap tools
From: MadHat <madhat () unspecific com>
Date: Sat, 6 Dec 2003 21:16:15 -0600

I have the responsibility of monitoring a large number of IPs for security issues. One of the most important things for me was to know what was listening where and of course nmap is the only real solution. The problem was that my boss wanted me to be able to generate a report of how many new ports were opened in the last 24 hours, how many new hosts in the past 24 hours, or even how many hosts we have live that are Internet facing or web servers, etc...

I looked at a lot of the tools out there. Many of them are great, but did not scale to the size I needed. I ended up writing some simple scripts in perl to do what I needed to do.


To give an example of what I am doing, I scan 80K+ IPs spread around the world in about 10 hours from a single, centralized host at this time. Of these, about 20K+ are accessible from the general Internet. I do basic scans of the registered ports and do an OS fingerprint. I have not added Version fingerprinting as of yet.

There are 2 main scripts I use daily, being the nmap-wrapper (which is really in constant flux trying to find the best configuration) and nmap-diff.

nmap-wrapper runs X number of nmap processes in parallel (I currently use 32). Each process only scans a single host for a few reasons. The data is taken back into the script and log files are created. Each class C has it's own log file per day. Not the best solution, but the easiest and most efficient I could find in a hurry. ( I have looked into the nmap-sql, but they don't seem to scale well for me. But I am still looking.) I scan a single host at a time because I want to know exactly when each host is being scanned (the start and stop time), but this has been fixed in later versions of nmap and verbose outputs. I also used a single host this way because, at least in older versions of nmap, I had issues with timing. It seemed that when trying to do it all at once, if it hit a block of hosts not responding, it would take forever to time out (which is where a lot of my tweaking has been) and not finish in a timely manner. The way nmap wrapper is written if one process hangs, it keeps going, for as one process ends it launches another and tries to keep X processes running at all times.

nnmap-diff is run every day after the nmap-wrapper is completed (see website for example). It does a host-by-host diff from the previous day (by default it is from 48 hours ago, but you can specify how many days back the base should be) and will gen a basic report of what changed (more specifically, by default it only shows what was added and not was went away). You can also only look for changes on specific ports, or specific IPs or partial IP matching. It also has the ability to mail the output to an email address, but I usually just pipe it to mail ;)

Using these I can detect within 24 hours of when a firewall hole is bad or a host has some unauthorized app listening on a non-standard port, etc... in theory anyway and I have detected some good stuff.

There are a couple of other scripts up there as well.

nmap-search, which I have sent to the list before (years ago when I wrote it), which allows you to search a single nmap -oG output based on a few different things.

nmap-report, which is similar to the above, but works on the flat file system generated by nmap-wrapper. I can tell it to give me all the hosts that have port 80 open, whether or not to include the DNS entry found by nmap at the time of the run, whether or not to report all the details found on hosts that match, match on OS, or partial IP (was going to be subnet, but I have not had a chance to fix it), or just search on a specific IP. It also allows you to specify how many day ago to search. The default searches yesterdays DB, but you can tell it to go back 3 days (-b3) and look for an answer then.

I have been using these for about a year and have had good success thus far. I am about to start rewriting some of them and recently had the opportunity to speak to Fyodor about these and some ideas I had and he recommended I pass them on here as other may want to check them out. So here it is....


Feedback is welcome as always.

MadHat (at) Unspecific.com

For help using this (nmap-dev) mailing list, send a blank email to nmap-dev-help () insecure org . List archive: http://seclists.org

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]