Nmap Development: [PATCH] Comm is tokin' and passing to HTTP

[Kris Katterjohn <katterjohn () gmail com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey everyone,

I've added a few new options to the Comm NSElib which I liked while looking at
the HTTP lib.  These in-turn allow Comm to be used as a back-end for HTTP.

Comm can now take four new options.  The first two are simple:

* conntimeout is for specifying a timeout only for the socket connect()
* reqtimeout is for specifying a timeout only for the receive*()s and send()s

The next two add the most new functionality:

* buf is for specifying a delimiter for parsing the data into tokens with the
lesser-used receive_buf() function.  This is used at the same level as the
"lines" and "bytes" options, but instead of limiting the amount of returned
data, it returns a table of all of the tokenized data (element=token).

* bufkeep is for specifying whether or not to keep the delimiter in the
returned data.  It defaults to false.  This means Comm uses receive_buf()
itself rather than the stdnse make_buffer() function which HTTP used.

I've attached a patch which adds these options to Comm and uses it in HTTP.

I've tested it against quite a few machines, with chunked encoding and not,
and it all works fine for me.

Please let me know what you think.

Thanks,
Kris Katterjohn

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIVAwUBSMS91v9K37xXYl36AQLN1g/+PaVIwJCCERtihTL5/rCAxWa21I4XHEoZ
x28Nt0QHC41j0nLUtxRCc7O0xwMhaUcdG5Fxfl6dip11lnruz38ZtSBUXwc16ptf
v7dFlDdeJiNnVtfC6zaxsH0/dUyoo/78o4iGTl/gCIAQhvpNy9vwyQLGHEFmSRPz
Xqa8Lyl+H9nWFeMpKhlY4knTpj49K6jvEbY1F1wxGG+gUY+aEgl1NaW8LOEompDL
KGuIO6kfpG1Iu7+TswVYMy4KIVTF4DAaVE1MJ2g0VIVEOtAIN6GGJhfcI+H9eivA
JQx1aTOyhOCE0/nH0vHDVD6YaZGHG3T0z9gombh57+rtkA4Uh5Eik4oYNXVcT7Dt
uWDMRJ/KePPJ2FXaU4bKhOkD2TpKb6XOBSn6hb4OOmAfPHNtBgQDWZj9uCw6JoVI
MUNFIMR8gbS5d6CW/LZDV+epner/Zb9pN7TEkTG+LyF2uuy7IS4ZBy1ELbJTBl7R
sTTSCRulffbzYsUwiFTHny4W2d83KpNZXbB8O/HvboCnKiBrgFjbZEdqwpPxm9LI
J5V8CXT2mrfHP19dqS9Rc4ky6m/3i1RlogNWjBLjU0IQOaAA2gzYj5e8WvvPLcqL
attLgWScm9pbtIoe2RvmF/t0PvmDgg51TSvbjp/88y95jLJwAUzIqbN3/Kiy1xmY
l0YPiLyWHLo=
=EK8N
-----END PGP SIGNATURE-----

Index: nselib/http.lua
===================================================================
--- nselib/http.lua     (revision 10021)
+++ nselib/http.lua     (working copy)
@@ -14,7 +14,7 @@
 
 module(... or "http",package.seeall)
 
-require 'stdnse'
+require 'comm'
 require 'url'
 
 --
@@ -125,34 +125,29 @@
   end
 
   local result = {status=nil,header={},body=""}
-  local socket = nmap.new_socket()
-  local default_timeout = {}
-  if options.timeout then
-    socket:set_timeout( options.timeout )
-  else
-    default_timeout = get_default_timeout( nmap.timing_level() )
-    socket:set_timeout( default_timeout.connect )
-  end
-  if not socket:connect( host, port, protocol ) then
+  local cto, rto
+
+  -- Set connect and request timeouts
+  cto = options.timeout or get_default_timeout(nmap.timing_level()).connect
+  rto = options.timeout or get_default_timeout(nmap.timing_level()).request
+
+  -- Comm options now
+  options = {proto=protocol, conntimeout=cto, reqtimeout=rto, buf="\r\n"}
+
+  local status, response = comm.exchange(host, port, data, options)
+
+  if not status then
     return result
   end
-  if not options.timeout then
-    socket:set_timeout( default_timeout.request )
-  end
-  if not socket:send( data ) then
-    return result
-  end
 
-  local buffer = stdnse.make_buffer( socket, "\r\n" )
-
   local line, _
   local header, body = {}, {}
 
-  -- header loop
-  while true do
-    line = buffer()
-    if (not line or line == "") then break end
-    table.insert(header,line)
+  -- Copy a separate header, removing it from response
+  while #response > 0 do
+    line = table.remove(response, 1)
+    if line == "" then break end
+    table.insert(header, line)
   end
 
   -- build nicer table for header
@@ -189,30 +184,25 @@
     -- if the server used chunked encoding we have to 'dechunk' the answer
     local counter, chunk_size
     counter = 0; chunk_size = 0
-    while true do
+    while #response > 0 do
       if counter >= chunk_size then
         counter = 0
-        chunk_size = tonumber( buffer(), 16 )
-        if chunk_size == 0 or not chunk_size then break end
+        chunk_size = tonumber( table.remove(response, 1), 16 )
+        if chunk_size == 0 or not chunk_size or #response == 0 then break end
       end
-      line = buffer()
-      if not line then break end
+      local line = table.remove(response, 1)
       counter = counter + #line + 2
       table.insert(body,line)
     end
   else
-    while true do
-      line = buffer()
-      if not line then break end
-      table.insert(body,line)
+    for _, line in ipairs( response ) do
+      table.insert(body, line)
     end
   end
 
-  socket:close()
   result.body = table.concat( body, "\r\n" )
 
   return result
-
 end
 
 get_default_timeout = function( nmap_timing )
Index: nselib/comm.lua
===================================================================
--- nselib/comm.lua     (revision 10021)
+++ nselib/comm.lua     (working copy)
@@ -5,12 +5,18 @@
 -- via nmap.new_try().\n
 -- \n
 -- These functions can all be passed a table of options, but it's not
--- required. The relevant indexes for this table are bytes, lines, proto
--- and timeout. bytes is used to provide the minimum number of bytes required
--- for a read. lines does the same, but for the minimum number of lines.
--- proto is used to set the protocol to communicate with, defaulting to
--- "tcp" if not provided. timeout is used to set the socket timeout (see
--- the socket function set_timeout() for details). 
+-- required. The relevant indexes for this table are bytes, lines, buf,
+-- bufkeep, proto, conntimeout, reqtimeout and timeout. bytes is used to
+-- provide the minimum number of bytes required for a read. lines does the
+-- same, but for the minimum number of lines. buf is used to set a delimiter
+-- for which to separate the data into tokens with, and causes the functions
+-- to return a table filled with these tokens as separate elements. bufkeep
+-- is a boolean value specifying whether or not to keep the delimiter in the
+-- returned buf data or not. proto is used to set the protocol to communicate
+-- with, defaulting to "tcp" if not provided. timeout is used to set the socket
+-- timeout (see the socket function set_timeout() for details). For specifying
+-- only the connect() timeout, you can use conntimeout.  For specifying just
+-- the receive*() and send() timeouts, you can use reqtimeout.
 -- @author Kris Katterjohn 04/2008
 
 module(... or "comm", package.seeall)
@@ -33,18 +39,9 @@
 -- they return is either the response from the host, or the error message
 -- from one of the previous calls (connect, send, receive*).
 --
--- These functions can be passed a table of options with the following keys:
+-- If no "lines", "bytes" or "buf" option is specified, the functions attempt
+-- to read as many bytes as possible.
 --
---   bytes: Specifies the minimum amount of bytes are to be read from the host
---   lines: Specifies the minimum amount of lines are to be read from the host
---   proto: Specifies the protocol to be used with the connect() call
---   timeout: Sets the socket's timeout with nmap.set_timeout()
---
--- If neither lines nor bytes are specified, the calls attempt to read as many
--- bytes as possible.  If only bytes is specified, then it only tries to read
--- that many bytes.  Likewise, it only lines if specified, then it only tries
--- to read that many lines.  If they're both specified, the lines value is used.
---
 ------
 
 -- Makes sure that opts exists and the default proto is there
@@ -74,21 +71,37 @@
 
        local sock = nmap.new_socket()
 
-       if opts.timeout then
-               sock:set_timeout(opts.timeout)
+       if not opts.conntimeout then
+               opts.conntimeout = opts.timeout
        end
 
+       if opts.conntimeout then
+               sock:set_timeout(opts.conntimeout)
+       end
+
        local status, err = sock:connect(target, port.number, opts.proto)
 
        if not status then
                return status, err
        end
 
+       if not opts.reqtimeout then
+               opts.reqtimeout = opts.timeout
+       end
+
+       if opts.reqtimeout then
+               sock:set_timeout(opts.reqtimeout)
+       end
+
        -- If nothing is given, specify bytes=1 so NSE reads everything
-       if not opts.lines and not opts.bytes then
+       if not opts.lines and not opts.bytes and not opts.buf then
                opts.bytes = 1
        end
 
+       if opts.bufkeep == nil then
+               opts.bufkeep = false
+       end
+
        return true, sock
 end
 
@@ -98,6 +111,21 @@
        if opts.lines then
                status, response = sock:receive_lines(opts.lines)
                return status, response
+       elseif opts.buf then
+               local token
+
+               response = {}
+
+               -- Start tokin' some tokens
+               while true do
+                       status, token = sock:receive_buf(opts.buf, opts.bufkeep)
+                       if not status then
+                               break
+                       end
+                       table.insert(response, token)
+               end
+
+               return (#response > 0), response
        end
 
        status, response = sock:receive_bytes(opts.bytes)
Index: docs/scripting.xml
===================================================================
--- docs/scripting.xml  (revision 10021)
+++ docs/scripting.xml  (working copy)
@@ -1650,13 +1650,22 @@
        <para>
        These functions can all be passed a table of options, but it's not required.
        The relevant indexes for this table are <literal>bytes</literal>, <literal>lines</literal>,
-       <literal>proto</literal> and <literal>timeout</literal>.  <literal>bytes</literal>
-       is used to provide the minimum number of bytes required for a read.  <literal>lines</literal>
-       does the same, but for the minimum number of lines.  If neither are provided, these
-       functions attempt to read as many bytes as are available.  <literal>proto</literal>
-       is used to set the protocol to communicate with, defaulting to <literal>"tcp"</literal> if not provided.
+       <literal>buf</literal>, <literal>bufkeep</literal>, <literal>proto</literal>,
+       <literal>conntimeout</literal>, <literal>reqtimeout</literal> and <literal>timeout</literal>.
+       <literal>bytes</literal> is used to provide the minimum number of bytes required for a read.
+       <literal>lines</literal> does the same, but for the minimum number of lines.  <literal>buf</literal>
+       is used to specify a delimiter with which to separate the data returned.  Using
+       <literal>buf</literal> returns all of the data, but as a table with the delimited tokens as
+       elements.  <literal>bufkeep</literal> is used to specify whether or not to keep the delimiter
+       in the table elements (it defaults to false).  If none of <literal>lines</literal>,
+       <literal>bytes</literal> or <literal>buf</literal> options are provided, these functions
+       attempt to read as many bytes as are available.  <literal>proto</literal> is used to set
+       the protocol to communicate with, defaulting to <literal>"tcp"</literal> if not provided.
        <literal>timeout</literal> is used to set the socket timeout (see the socket function
-       <literal>set_timeout()</literal> for details).
+       <literal>set_timeout()</literal> for details). For specifying only the <literal>connect()</literal>
+       timeout, you can use <literal>conntimeout</literal>.  For specifying only the only the
+       <literal>receive*()</literal> and <literal>send()</literal> timeouts, you can use
+       <literal>reqtimeout</literal>.
        </para>
 
        <variablelist>

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org