Re: Error in http.lua's chunked encoding

[David Fifield <david () bamsoftware com>]

On Tue, Aug 18, 2009 at 01:03:59AM -0300, Joao Correa wrote:
> The problem happens because the server is returning an extra \n. Do
> you know if this is a usual thing? Maybe under the conditions of
> making HEAD requests under a pipeline to a server that responds with
> chunked encoding, this is the default way of answering.

The error is not in the server, but in the splitResults function. It
wrongly adds an extra newline to the last response.

  response = stdnse.strsplit("\n", response)

  for _, line in ipairs( response or {} ) do
    if(string.match(line, "HTTP/%d\.%d %d+")) then
      i = i + 1
      results[i] = ""
    end
    results[i] = results[i] .. line .. "\n"
  end

Breaking the response stream on anything that looks like a Status-Line
is really wrong. What if a response contains "HTTP/1.1 200" in the body?
What if the body doesn't end in a newline, so that the Status-Line of
the next response doesn't come at the beginning of a line? Responses to
pipelined requests are supposed to either have a Content-Length or use
the chunked Transfer-Encoding, both of which allow you to find the
length of the body and the start of the next response. The get_chunks
iterator returns the offset after the last chunk so you can start
parsing the next response there.

Here is a quick and dirty fix for the extra newline problem. It still
has the other problems I mentioned above.

local function splitResults( response )
  local results = {}
  local line
  local i, offset

  i = 1
  while true do
    offset = string.find(response, "HTTP/%d\.%d %d+", i + 1)
    if offset then
      results[#results + 1] = string.sub(response, i, offset - 1)
    else
      results[#results + 1] = string.sub(response, i)
      break
    end
    i = offset
  end
  return results
end

> > > > ./nselib/http.lua:164: Didn't find CRLF after chunk-size [
> > > > chunk-extension ]
> > > > at position 2; got "OF\
> > > > ".
> > > > stack traceback:
> > > >        [C]: in function 'error'
> > > >        ./nselib/http.lua:164: in function '(for generator)'
> > > >        ./nselib/http.lua:834: in function 'parseResult'
> > > >        ./nselib/http.lua:682: in function 'pipeline'
> > > >        ./scripts/http-enum.nse:169: in
> > > > function<./scripts/http-enum.nse:42>
> > > >        (tail call): ?

The above error is the result of a different bug that was fixed by João
in a later patch. Responses to pipelined requests are read with this
loop:

    response_raw = ""
    while recv_status do
      recv_status, response_tmp = socket:receive()
      response_raw = response_raw .. response_tmp
    end

When the last read hits EOF, the return value of socket:receive() is
(false, "EOF"). The loop should exit at this point, but it wrongly
appends the "EOF" to the body of the final response. That's why you got
the error referring to "EOF\n"; 'E' happens to be a hex digit so it
looks like the start of a chunk, but what follows the hex digit is
supposed to be CRLF.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org