mailing list archives
[NSE] ideas for http library
From: Patrick Donnelly <batrick () batbytes com>
Date: Thu, 27 Oct 2011 02:42:53 -0400
I promised a while back I'd make a post about this earlier this Summer
and just now remembered:
We were discussing pipelining for the http library and how it doesn't
currently use the caching mechanism of http.get. I brought up how it'd
be useful if pipelining were transparent and automatic via http.get
and the rest of the current pipelining API were thrown out.
Essentially, the http library would have separate worker threads (one
thread for each host) which would concatenate requests for URIs to a
single host via pipelining and give back the response to the script
which requested it via some sort of callback (or similar design). The
advantages of this is that we have only one active connection to an
HTTP server which reduces load, improves overall performance, and
increases parallelism (more scripts can do HTTP requests at once).
Some things to consider:
o The worker thread would "belong" to the first script which made the
request. I believe currently that worker thread would function
normally despite that script possibly finishing before its work is
done. Perhaps there should be a separate "library thread" function
which would have separate ownership mechanisms.
o http.get should maybe block in certain circumstances e.g. when the
pipeline "queue" is full. That way a script doesn't make thousands of
requests and then wait for callbacks to return results.
o Other http functions should also work in the same pipeline, e.g. http.head.
o Obviously, the cache should function in the new system.
- Patrick Donnelly
Sent through the nmap-dev mailing list
Archived at http://seclists.org/nmap-dev/
- [NSE] ideas for http library Patrick Donnelly (Oct 27)