Re: Pipelining and flowpinning

[Martin Roesch <roesch () sourcefire com>]

Hi Jonathan,

We've been aware of the papers on flow-pinning and pipelining since
they came out back in 2006, they were definitely interesting and we've
taken a look at them and the ideas that they put forth internally and
even looked at them within the context of Snort 2.x and 3.0.  They are
a little bit out of date now (the technologies and ideas have
advanced) but they make good points.

I think some of the multithreading hype that's been going on lately
needs to be examined in context.  To my mind the reason you'd want to
multi-thread an application like Snort is for the purposes of:

- A shared management interface to multiple analysis threads.
- The ability to share information about the environment you're
protecting between threads.
- Load balancing across detection threads with a common configuration.

The idea in Snort 3.0/SnortSP is to have common shared data structures
and memory that allow things like network discovery capabilities to
share info in real-time with network control capabilities (e.g.
auto-tuning IPS).  Load spreading across CPU cores can be accomplished
just as easily with a load balancing front-end on multiple processes
as in a multithreaded design, multithreading for the sake of
multithreading doesn't really buy you performance and burdens you with
concurrency and locking overhead which is not computationally cheap
with pthreads.  Multithreading the Snort analysis model is not going
to produce any dramatic performance advantages over an intelligently
load balanced and properly configured Snort 2.x engine.  If you want
to change the performance dynamic you should examine alternate
detection strategies that optimize full (7 layer) protocol processing
performance.

We've seen well over 1Gbps of performance per core on the current
Snort 2.x architecture running a well formed ruleset and using our
software-based load-balancing capabilities.  I still think that
multithreading is the way to go for the Snort 3.0 architecture but
it's primarily for the 3 reasons I mentioned above.  We're going to
continue to use kernel-based load balancing across CPU locked threads
(i.e. the same model we use now) to maximize he performance of single
configuration instances in the new model but we're also going to be
trying new detection models as well.  Stay tuned!


Marty



On Thu, May 20, 2010 at 5:53 AM, Jonathan Saint-Léger
<tan.saintleger () gmail com> wrote:
> Hi,
>
> I am working on Snort optimization and I am aware that Snort 2.X versions don't handle multi-threading, and it should 
> be one of the big steps for Snort 3.0.
> However, I saw some very interesting papers: http://download.intel.com/technology/advanced_comm/31431201.pdf and 
> http://download.intel.com/technology/advanced_comm/31156601.pdf from intel about pipelining and flowpinning.
> Does anybody know if it Is possible to reproduce the pipelining and flowpinning trick on a Snort 2.X release?? Or 
> were these tests done with a beta Snort 3.0 release?
>
>
> Regards,
> Jonathan
>
> ------------------------------------------------------------------------------
>
>
> _______________________________________________
> Snort-devel mailing list
> Snort-devel () lists sourceforge net
> https://lists.sourceforge.net/lists/listinfo/snort-devel



--
Martin Roesch - Founder/CTO, Sourcefire Inc. - +1-410-290-1616
Sourcefire - Security for the Real World - http://www.sourcefire.com
Snort: Open Source IDP - http://www.snort.org

------------------------------------------------------------------------------

_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel