Proposal for adding buffer to Nsock read functions.

[venkat sanaka <venkatsanaka () gmail com>]

Hi all

I have finished writing the proposal for adding a buffer to Nsock read
functions.
I would like to receive comments, suggestions and any alternatives for
implementing this idea to Nsock library.


Regards
Venkat
............................................................................................................................
                               Nsock Buffer Management For Read Functions.

Current Implementation:
At present Nsock API provides three different functions to read, namely
nsock_readlines,nsock_readbytes and nsock_read.

nsock_readlines: This reads atleast n lines (terminated with \n, which
of course includes \r\n), or read up until EOF or timeout, whichever
comes first.

nsock_readbytes: This works similar to nsock_readlines except that it
reads atleast n bytes instead of n lines.

nsock_read: This is like the read system call. You get some amount of
data, but you don't have control over it.

Problem with current implementation:
The main problem of current implementation is lack of buffer to read
functions in Nsock. I will explain the need of buffer to Nsock read
functions by stating some situations, where we have difficulties in
using the current Nsock read API.

1. Ncat should send one line at a time in when --delay option is given.
   For this Ncat should read data from the STDIN and store it in a
   buffer. Then one line at a time is read from the buffer and write
   that line to the socket for sending. This is not possible now, as
   there is no read function in Nsock which gives exactly one line of
   read data and store the rest of the read data in buffer for sending
   later, in the same way as above.

2. The size of the socks4 response is "7" bytes.
   So we need to read exactly "7" bytes from the buffer for processing
   the socks4 response for validity and if socks4 response is valid,rest
   of the buffered read data is written to the STDOUT.SO, buffer is
   needed here for storing the extra bytes (other than 7) from being
   lost.

3. Similar problems discussed in this post.
   http://seclists.org/nmap-dev/2009/q2/0673.html
   Buffering problems in ssh2.lua.This problem occured when an SSH-2 packet is
   split across two TCP packets.Then the read only gets the first half,
   and subsequent operations on the partial packet lead to the crash.
   So we have to do the operations like parsing or checking the validity
   of the packet etc; only when we received the exact amount of data needed.
   Untill then, the data read should be buffered.


4. When parsing a http header,one line at a time is read looking for
   "\r\n" or "\n". Suppose we received a partial http response like the
   below one in the first network read().

   HTTP/1.0 200 OK \r\n
   Accept:

   Here one line at a time should be read from the buffer while parsing
   and  store the incomplete line ("Accept: ") in the buffer
   from being lost, which is parsed with the received data from the subsequent
   network read()s.

   The above exmaples show the importance of having a buffer to read
   functions and how it can be used in some situations where we need
   exact amount of data to be read from network/STDIN.

   These applications can work around this by implementing their own
   buffer (as NSE does), but it will be more convenient to be built into
   the Nsock library itself. So the nsock_readlines and nsock_readbytes
   functions should give exactly specified no. of lines and bytes respectively
   and  should also store the rest of the data from being lost in a buffer for
   later operations. Both of these behaviours are lacking in Nsock API.

Existing similar solutions:
1. nmap.receive_buf in NSE which reads upto a particular
   delimiter (pattern/separator).
   http://nmap.org/nsedoc/modules/nmap.html#receive_buf
2. socket_buffer in http.h, which is a stateful buffer. This buffer type
   is used in socket_buffer_read, socket_buffer_readline,
   socket_buffer_readcount and socket_buffer_remainder functions of
   http.c to return exactly the specified no. of lines and bytes and
   store the remaining data in the same buffer.

Proposed implementation:
The basic idea of this proposal is to add buffer to Nsock read functions
without changing much in the usage of the Nsock API.

A stateful buffer similar to socket_buffer in http.h, which should have
state information like no of bytes read upto and the no of bytes
remaining in the buffer  etc; is proposed to use for Nsock read
functions too.

This is how Nsock read functions should work after adding the buffer to them.
For example:
* I use nsock_read_bytes(7) because I want exactly 7 bytes for SOCKS4.
* The handler for read_bytes does a recv and gets whatever is on the
  socket, let's say it gets 36 bytes.
* But it returns only 7 bytes and store the remaining 29 bytes in a
  buffer. (Currently, nsock_read_bytes will return all 36 bytes even
  though 7 is specified while calling.)
* Next if nsock_read_bytes(15) is called, and instead of reading from
  the network, it would immediately return 15 bytes from the buffer.
* If the buffer doesn't contain requested no. of bytes, then it would do
  as many as recvs needed to get a data of requested size.
* nsock_read_lines() also works in a similar way as above, returning the
  requested no. of lines.

Buffer:A new stateful buffer will be added in msiod structure.

Note: For rest of the proposal,the usage of word "buffer" refers to
"read_buf" added in msiod.

New Additions:

  Data Structure              Member

1. mspool                     int buffered_read_events
   No of read events whose buffer is not empty (means some data is
   buffered).

2. read_buf(new structure)    char *start,*end
                              char buffer[BUFSIZ]
                              int nbytes    /* No. of lines in the buffer */
                              int nlines    /* No. of bytes in the buffer */
   This is a new stateful buffer added to msiod. Nsock read functions
   stores the total data read from that iod in this buffer instead
   of filespace buffer present in msevent. Only requested no. of bytes
   or lines is moved to the filespace buffer of that event.
   For eg: We want 10 bytes of data and we used nsock_readbytes(10)
   function for this. But the no. of bytes read from the network/STDIN is
   20, now this 20 bytes are stored in read_buf of msiod and out of this
   20 bytes, the only requested 10 bytes are added to filespace buffer of
   that read event, which we can get using nse_readbuf() function.

   Finally to say, the filespace contains the data that is going to be
   returned to the user. The iod buffer contains the data that is being
   kept until the next read.

3. msiod                      struct read_buf *buf;
   If an read event whose buffer has some left over data,is going to be
   deleted, then it stores that remaining data in this buffer of its
   msiod,so that this data can be passed to subsequent read events
   created for that msiod.

4. event_lists                gh_list buf_read_events
   New list which contains the read events with buffered data,so that we
   can handle them in this order: connect => buf_read =>read=>write=>timer


Changes in Behaviour of functions:

1. handle_read_result(): Store the data in read_buf of msiod instead of
   filespace buffer of msevent and add specified no of bytes/lines
   to filespace buffer.
2. nsock_readbytes() and nsock_readlines(): After creating a new read
   event in these functions, we check if read_buf of msiod associated
   with this new event is not empty (has some data in it from previous
   read events of that iod.) and also look that buffer has enough
   no. of bytes/lines requested, by checking the value of read_buf->nbytes
   or read_buf->nlines.
   If yes, then no need of reading from the network/STDIN.So,
   * The event is added to buf_read_events instead of read_events, using
     gh_list_append.
   * Increment the count of buffered_read_events in mspool.
3. nsock_loop(): Check for any buffered_read_events in mspool before
   calling wait_for_events().

Process Flow:

 nsock_readbytes(nbytes)/nsock_readlines(nlines)
       |
       |
 msevent_new
       |
       |
 nsp_add_event
       |
       |
gh_list_remove() from read_events and gh_list_append() to buf_read_events.
 (The last one happens only when the the read buffer of iod
  associated with that read event is not empty).

The above process takes place in nsock_readbytes(nbytes)
and nsock_readlines(nlines) functions.

The below one is the process flow in the Nsock loop.

nsock_loop()
       |
       |
  Check for any buffered_read_events of mspool
       |                                        |
       |yes                                   |No
       |                                        |
       |                                  wait_for_events(select)
 iterate_through_events<------------------------|
       |
       |
 handle_read_result
       |
       |
  do_actual_read


Test Program:

.......................................................................................................................................................

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org