Home page logo

nmap-dev logo Nmap Development mailing list archives

Re: [nmap-svn] r28585 - nmap/todo
From: Patrick Donnelly <batrick () batbytes com>
Date: Wed, 16 May 2012 00:41:50 -0400

On Tue, May 15, 2012 at 5:16 PM,  <commit-mailer () insecure org> wrote:
Author: fyodor
Date: Tue May 15 14:16:49 2012
New Revision: 28585

Add a todo entry about fixing NSE pipelining


Modified: nmap/todo/nmap.txt
--- nmap/todo/nmap.txt  (original)
+++ nmap/todo/nmap.txt  Tue May 15 14:16:49 2012
@@ -123,6 +123,27 @@
  o Review NSE-based port scanning and RST idle scan.
    http://seclists.org/nmap-dev/2011/q2/307. [Henri and Hani?]

+o Investigate why http pipelining so often doesn't work in NSE
+  scripts, and often NSE ends up reverting to one request at a time.
+  Scripts may not be using it correctly, and also we wish it were more
+  transparent and there wasn't this big API divide between pipeline
+  and non-pipeline.  We just want it send requests as fast as it can,
+  and get a callback when there's a response.  Maybe the http library
+  buffers them, or pipelines them, or blocks the http.get call until
+  there's more room.  It just seems to always degenerate to 1 request
+  at a time.  For example:
+  sudo nmap --script=http-enum bamsoftware.com -p80 -d2
+  quickly (within a few seconds) gives:
+    NSE: http-enum: Searching for entries under path '' (change with 'http-enum.basepath' argument)
+    NSE: Total number of pipelined requests: 2081
+    NSE: Number of requests allowed by pipeline: 100
+    NSE: Received only 41 of 100 expected responses.
+    Decreasing max pipelined requests to 41.
+    NSE: Received only 1 of 41 expected responses.
+    Decreasing max pipelined requests to 1.
+  100 may a wildly high number of requests to attempt to pipeline.
+  And then something else probably goes wrong after it decides 41 is okay.

My earlier thoughts on the subject (for those reading nmap-svn and curious):


I think callbacks in general will be, from a programmatic perspective,
inconsistent with the rest of our libraries. In particular, the nsock
binding does not expose the callback mechanism to scripts. I would
like to see a design that avoids callbacks.

One possibility is to keep http.get as a "blocking" call but scripts
can achieve http pipelining by having multiple workers. I don't think
this is too unreasonable as we don't want more than ~5 pipelined
requests anyway. So, the http library will automatically pipeline
requests to the same host coming from multiple threads but one thread
can only have one request pending at a time.

- Patrick Donnelly
Sent through the nmap-dev mailing list
Archived at http://seclists.org/nmap-dev/

  By Date           By Thread  

Current thread:
  • Re: [nmap-svn] r28585 - nmap/todo Patrick Donnelly (May 16)
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]