mailing list archives
Re: Nsock read buffer
From: David Fifield <david () bamsoftware com>
Date: Tue, 11 Jan 2011 20:57:50 -0800
On Mon, Nov 15, 2010 at 10:23:15PM -0800, David Fifield wrote:
I started implementing this in the branch
svn co --username guest --password "" svn://svn.insecure.org/nmap-exp/david/nmap-readbuf
I attached a script that shows how it works. nsock_readbytes and
nsock_readlines (and hence nmap.receive_bytes and nmap.receive_lines)
return exactly what you ask for, not more and not less. If there are not
enough bytes available, the functions return an error and the bytes stay
in the buffer for the next read. If there are more than enough bytes
available, only the first n bytes or lines are returned, and what's left
over remains in the buffer.
I think that the behavior of nsock_readbytes and nsock_readlines in my
branch is just write. My question is, how should nsock_read work?
nsock_read is used to mean both of these two things:
1. Keep reading until timeout or EOF, and return everything.
2. Return me some small chunk of data, up to one buffer's worth, but I
don't care about the exact number.
In practice, nsock_receive does (1), except that there a built-in limit
of 589823 bytes, after which the read always finishes. (Look for the
comment "spews and spews data" in nsock_core.c.) When people want (2),
they have been using nsock_readbytes(1), which only does one recv and
only returns up to a relatively small fixed-size buffer (like recv).
My personal feeling is the nsock_read should work like (2). But we still
want a function that works like (1) and returns a big block of data all
at once. Service scan does this for example. But even service scan has
no use for 589823 bytes; it seems like the caller should be able to set
this limit. "As many bytes as are available, but not more than X." This
is pretty close to how nsock_readbytes works now, in that it quits
reading once enough bytes are available, except that you can still get
more than you asked for (up to one fixed-size buffer's worth more).
I've been going back and forth over the API for these read functions.
I've identified four use cases:
1. Return everything left on the socket (looped reading), with a
sanity limit on bytes.
2. Return whatever is immediately available on the socket.
3. Return an exact number of bytes.
4. Return an exact number of lines, with a sanity limit on bytes.
In the branch, nsock_readbytes and nsock_readlines do (3) and (4).
nsock_read does (2), and there's no built-in way to do (1).
I updated some scripts that were, for example, using
socket:receive_bytes(1), to use socket:receive() instead. However I
can't guarantee that this is correct in every case.
Another question is what to do with sanity byte limits. We can't just
read forever if someone keeps writing without sending a newline. Current
nsock_readlines already has such a limit, only it is an undocumented,
unspecified internal constant. If there's no newline you get back a
partial line. An alternative is just to return an error and not violate
the contract that we will return only the requested number of lines.
Some of the migration is tricky. Here is an example from domcom-cmd.nse.
The call to receive_lines(1) is expecting to get more than one line. It
looks like it's expecting a line with "BeginData" and one with
local status, line = socket:receive_lines(1)
if ( not(status) ) then return false, "Failed to read line" end
lines = stdnse.strsplit( "\n", line )
for _, line in ipairs( lines ) do
if ( not(line:match("BeginData")) and not(line:match("EndData")) ) then
Strictly speaking, this code is incorrect. There's no reason why
receive_lines has to return everything on the socket. It just happens
that line endings fall on packet boundaries so no half-lines are
returned. Since the new receive_lines(1) will always return exactly one
line, this has to be rewritten. What's wanted here is an nsock_read that
does option (1) above. But then it becomes obvious that the code is
relying on an integer number of lines being returned, and you have to
write an explicit loop with receive_lines(1). This is what I have done
but it's more code, and frustrating for a project that was supposed to
Sent through the nmap-dev mailing list
Archived at http://seclists.org/nmap-dev/
- Re: Nsock read buffer David Fifield (Jan 12)