Re: Weird Crash - "WAITING_TO_RUNNING"

[Nathan <nathan.stocks () gmail com>]

On Wed, Nov 17, 2010 at 12:34 PM, David Fifield <david () bamsoftware com> wrote:
> On Mon, Nov 15, 2010 at 02:53:59PM -0700, Nathan wrote:
> > > > > On Tue, Nov 09, 2010 at 10:32:19AM -0800, David Fifield wrote:
> > > > > Nathan, please try out this nse_main.lua. It's has a quick and dirty
> > > > > modification that prevents the creation of more than 100 script threads
> > > > > at a time. Run the scan so that it creates lots of spurious open ports
> > > > > like before. It should not use up all your memory and should eventually
> > > > > finish.
> > > > >
> > > > > I think we will actually set the limit higher than 100 in practice.
> >
> > Okay, it didn't change the accuracy (we didn't expect it too), so it
> > still thought all 65k+ ports were open.  But it certainly limited RAM
> > usage and actually finished!
> >
> > It was using about 55MB RAM when it ended, and it took 5m23s -- a huge
> > improvement over using 4GB of RAM and crashing!
>
> I just committed this as r21084. I increased the limit from 100 to
> 1,000. I also made some code changes so please test it again and see if
> it works.
>
> Patrick D.: I defined a new file-level local variable CONCURRENCY_LIMIT
> in nse_main.lua. Is this the best place for it?

Okay, so I tested 2.36TEST2, which should be up to r21143, so it
should include your change from 100 to 1,000 concurrent service
detection threads.

=> The run with CONCURRENCY_LIMIT=100 BEFORE these changes took about
55MB RAM and 5m23s.
=> The run with CONCURRENCY_LIMIT=1000 AFTER these changes took about
95MB RAM and 14m16s

and then I edited nse_main.lua and lowered the limit to 100 to see if
that would restore the better performance, and got:

=> The run with CONCURRENCY_LIMIT=100 AFTER these changes took about
69MB RAM and 12m56s

So, assuming that ALL of the following are true:
- my test server is sane (it should be, I've pretty much only been
using it to test these nmap changes the last couple weeks)
- the Internet route between me and my target is pretty much the same
(no way to know, because I didn't take a traceroute)
- nothing drastic changed on the target itself (I don't think it has,
but who knows...)
- no other change to nmap affected this (beats me)

THEN

It appears that your latest changes affected things quite adversely,
and raising the concurrency limit to 1000 didn't help either.

I think your older version worked much faster and used much less RAM
and that we should revert to it.  I will use it on my servers, at
least, for the time being.

Thanks for all the awesome help--it's nice to participate in such an
active project.

~ Nathan
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/