mailing list archives
Re: [nmap-svn] r28585 - nmap/todo
From: Patrick Donnelly <batrick () batbytes com>
Date: Wed, 16 May 2012 00:41:50 -0400
On Tue, May 15, 2012 at 5:16 PM, <commit-mailer () insecure org> wrote:
Date: Tue May 15 14:16:49 2012
New Revision: 28585
Add a todo entry about fixing NSE pipelining
--- nmap/todo/nmap.txt (original)
+++ nmap/todo/nmap.txt Tue May 15 14:16:49 2012
@@ -123,6 +123,27 @@
o Review NSE-based port scanning and RST idle scan.
http://seclists.org/nmap-dev/2011/q2/307. [Henri and Hani?]
+o Investigate why http pipelining so often doesn't work in NSE
+ scripts, and often NSE ends up reverting to one request at a time.
+ Scripts may not be using it correctly, and also we wish it were more
+ transparent and there wasn't this big API divide between pipeline
+ and non-pipeline. We just want it send requests as fast as it can,
+ and get a callback when there's a response. Maybe the http library
+ buffers them, or pipelines them, or blocks the http.get call until
+ there's more room. It just seems to always degenerate to 1 request
+ at a time. For example:
+ sudo nmap --script=http-enum bamsoftware.com -p80 -d2
+ quickly (within a few seconds) gives:
+ NSE: http-enum: Searching for entries under path '' (change with 'http-enum.basepath' argument)
+ NSE: Total number of pipelined requests: 2081
+ NSE: Number of requests allowed by pipeline: 100
+ NSE: Received only 41 of 100 expected responses.
+ Decreasing max pipelined requests to 41.
+ NSE: Received only 1 of 41 expected responses.
+ Decreasing max pipelined requests to 1.
+ 100 may a wildly high number of requests to attempt to pipeline.
+ And then something else probably goes wrong after it decides 41 is okay.
My earlier thoughts on the subject (for those reading nmap-svn and curious):
I think callbacks in general will be, from a programmatic perspective,
inconsistent with the rest of our libraries. In particular, the nsock
binding does not expose the callback mechanism to scripts. I would
like to see a design that avoids callbacks.
One possibility is to keep http.get as a "blocking" call but scripts
can achieve http pipelining by having multiple workers. I don't think
this is too unreasonable as we don't want more than ~5 pipelined
requests anyway. So, the http library will automatically pipeline
requests to the same host coming from multiple threads but one thread
can only have one request pending at a time.
- Patrick Donnelly
Sent through the nmap-dev mailing list
Archived at http://seclists.org/nmap-dev/
- Re: [nmap-svn] r28585 - nmap/todo Patrick Donnelly (May 16)