Secure Coding mailing list archives
Positive impact of an SSG
From: list-spam at secureconsulting.net (Benjamin Tomhave)
Date: Wed, 11 Mar 2009 17:11:47 -0400
I think something significant has been excluded from this thread on BSIMM, and that's the role of SSF. If I am to understand correctly, SSF implementation was the experiment, and BSIMM was the resultant model for assessing (measuring) the maturity of software security programs being developed with SSF. Is that a fair read? In the end, the same problem exists here: development of a model based on data does not a validated model make. By way of Bayesian thinking, you can definitely create a viable model based on a small sampling (FAIR is based on this principle). However, you still have to go back and /independently/ validate that model by exercising it outside of the original data set. Does BSIMM appear to be a viable model for assessing the maturity of SSF-based software security programs? Sure. Has BSIMM been independently validated using a similar data set? No. Is BSIMM inherently reliant on SSF? Methinks so. Does BSIMM make universal sense for measuring *any* software security program? Unclear, but is questionable. Is SSF a de fact standard for building a software security program? Unclear. Anyway... fun debate! I fully appreciate the new work here, and I think it holds promise. The release of the initial draft is probably cause for a small celebration by the creators. However, until the model has been validated with independent data and has been shown to be generically applicable, I think there is still reason to hold back the really good bubbly. ;) cheers, -ben Brian Chess wrote:
Ben, Pravir,
Skepticism is healthy, but recognize that:
1) These criticisms would not be possible if we hadn?t explained how
BSIMM was created and offered up the origins of the data. If we simply
said ?here, we think this will probably work?, we could have created a
much more elegant model, and it would have been easy to brush all of the
?how did you validate it?? questions under the rug with lines like
?Trust us, we?re the experts!?
2) We published our data set. If you don?t like the model we created,
you can use our data to create your own. If you don?t think we came at
this the right way, you can conduct a better experiment, publish your
data, and demonstrate that ours is inferior. I hope that happens. It?s
how progress occurs in many other disciplines.
So then, is software security a solved problem? Of course not! There?s
plenty left to be done, and the landscape will be different next year.
We will have new dilemmas and we?ll be working under tighter
tolerances. We will need a constant stream of new and unproven ideas to
try out and report back on. So BSIMM isn?t game over, but in moving
from ?no supporting evidence? to ?based on the data?, we?ve raised the bar.
Ben, thanks for the DNS digging.
Brian
On 3/11/09 1:32 PM, "Benjamin Tomhave" <list-spam at secureconsulting.net>
wrote:
I think the celebration is a bit premature, for many of the reasons
Pravir just covered. I think that perhaps the problem we're having here
is that you've not really tested your results, nor have you iterated
through a 2nd time to reevaluate the working theory. If you were
approaching this scientifically, I think the process would look like
this (http://en.wikipedia.org/wiki/Scientific_method):
1. Use your experience: Consider the problem and try to make sense
of it. Look for previous explanations. If this is a new problem to you,
then move to step 2.
2. Form a conjecture: When nothing else is yet known, try to state
an explanation, to someone else, or to your notebook.
3. Deduce a prediction from that explanation: If you assume 2 is
true, what consequences follow?
4. Test : Look for the opposite of each consequence in order to
disprove 2. It is a logical error to seek 3 directly as proof of 2. This
error is called affirming the consequent.
I think what your missing, then, is at least step 4 as well as
reiterating through the process again (and possibly Step 3). It's a bit
abstract, perhaps, to rigidly apply the scientific method to this
program, but I think it's instructive to consider how you might do this.
BTW, your citation of the xkcd strip on causation vs correlation is
actually instructive here. You've developed a model based on correlation
without demonstrating causation at all. Not to get abstruse, but I don't
see that your model is properly supported or validated. In the end,
ironically, it seems to come down more to expert theories than empirical
evidence. A handful of experts studied 9 organizations and correlated
"highly successful" with "110 practices", but without properly defining
success, without generalizing the model to all types of organizations
(or without defining the scope), and without testing/validating the
model.
The good news is that you can now test the model. The bad news is that
you ("you" being the collective behind BSI-MM) probably should have
tested the model first before jumping straight to fanfare and hoopla. :)
In the end, I'm sure that BSI-MM will be a fine model, though the
questions will then be "can I implement it?" and "will it have
sustainable value on its own?" If the value of the model rests on
Cigital and Fortify pushing it into organizations by force (much as the
Big N, ISACA, and ITGI have tried to do with CObIT and valIT), then I
submit that it will encounter problems. It needs to be able to stand on
its own, properly validated, with inherent value through logical
implementation.
Which perhaps begs a question: is BSI-MM intended as an implementation
model to achieve better security in software development, or is it a
measurement tool for evaluating the current security maturity of
software development? A maturity model is typically used to measure
maturity, which means that someone has to then come along and provide
guidance on how to implement a program that can reach an optimal
measurement. (and mayhaps this would be a good time to get together with
Pravir to see if there's a way that you could both have winning game
plans)
BTW, when you get to the point of defining success, I would suggest
looking at FAIR (since they lean toward quantitative vs qualitative risk
assessment based on Bayesian statistics) as well as looking at the
concept of "risk resiliency" advocated, in particular, by BT. fwiw.
Anyway...
On whether the site is up or not, I think DNS is hosed for the domain...
I tried it from three locations (separate regions, separate providers)
and got the same results:
$ host bsi-mm.com
Host bsi-mm.com not found: 3(NXDOMAIN)
$ host bsi-mm.com
;; connection timed out; no servers could be reached
freeproxy.us also times out...
cheers,
-ben
Brian Chess wrote:
> Ben! Thank you! When you talk about sample size, it gives me hope
that
> we?re on the right track. We can either:
>
> 1) Use ideas that ?experts? theorize will work
> or
> 2) Gather empirical evidence to judge one idea against another.
>
> We in the security crowd often try to hide behind the need for secrecy,
> and that?s pushed us toward relying almost entirely on people who have
> nothing but rhetoric and personal reputation to stand on. BSIMM pretty
> well shows that, in 2009, we can do better. It?s a big step forward to
> collect data and then argue about what it means. I know it?s already
> made the rounds, but let?s have some XKCD to celebrate:
> http://xkcd.com/552/
>
> I think your question about defining success is an important one. We
> were loose about it in this first round, and I hope it?s something we
> can tighten up in our follow-on work. Here?s my thinking as of today:
> software security is not the goal, it?s one of the many things an
> organization needs to do in order to meet it?s objectives. We need to
> look at how a software security initiative (or lack thereof)
effects the
> organization?s ability to meet it?s objectives. This is going to be
> messy, but it?s either that or go back to making stuff up.
>
> BTW, I checked the BSIMM web site after I read your message. It worked
> for me. Try this?
> http://www.downforeveryoneorjustme.com/bsi-mm.com
>
> Brian
>
> On 3/11/09 10:48 AM, "Benjamin Tomhave"
<list-spam at secureconsulting.net>
> wrote:
>
> I think it's an interesting leap of faith. Statistically
speaking, 9 is
> a very small sample size. Thus, the proposed model will be viewed
> skeptically until it is validated with a much larger and more
diverse
> sample. Putting it another way, there's no way I can take this to a
> small or medium sized org and have them see immediate
relevance, because
> their first reaction is going to be "those are 9 large orgs
with lots of
> resources - we don't have that luxury."
>
> You quoted "we can say with confidence that these activities are
> commonly found in highly successful programs" - how do you define a
> "highly successful program"? What's the rule or metric? Is this
a rule
> or metric that can be genericized easily to all development teams?
>
> My concern is exactly what you speculate about... organizations are
> going to look at this and either try to tackle everything (and
fail) or
> decide there's too much to tackle (and quit). In my experience
working
> with maturity models as a consultant, very few people actually
> understand the concept. Folks are far more tuned-in to a PCI-like
> prescriptive method. Ironically, the PCI folks say the same
thing you
> did - that it's not meant to be prescriptive, that it's
supposed to be
> based on risk management practices - yet look how it's used.
>
> Maybe you've addressed this, but it doesn't sound like it. I'd
perhaps
> be better educated here if the web site wasn't down... ;)
>
> -ben
>
> Sammy Migues wrote:
> > Hi Pravir,
> >
> > Thanks for clarifying what you're positing. I'm not sure how
we could
> > have been more clear in the BSIMM text accompanying the
exposition of
> > the collective activities about the need to take this
information and
> > work it into your own culture (i.e., do "risk management").
As a few
> > examples:
> >
> > p. 3: "BSIMM is meant as a guide for building and evolving a
software
> > security initiative. As you will see when you familiarize
yourself
> > with the BSIMM activities, instilling software security into an
> > organization takes careful planning and always involves broad
> > organizational change. By clearly noting objectives and goals
and by
> > tracking practices with metrics tailored to your own
initiative, you
> > can methodically build software security in to your
organization?s
> > software development practices."
> >
> > p. 47: "Choosing which of the 110 BSIMM activities to adopt
and in
> > what order can be a challenge. We suggest creating a software
> > security initiative strategy and plan by focusing on goals and
> > objectives first and letting the activities select themselves.
> > Creating a timeline for rollout is often very useful. Of course
> > learning from experience is also a good strategy."
> >
> > p. 47: "Of the 110 possible activities in BSIMM, there are ten
> > activities that all of the nine programs we studied carry
out. Though
> > we can?t directly conclude that these ten activities are
necessary
> > for all software security initiatives, we can say with confidence
> > that these activities are commonly found in highly successful
> > programs. This suggests that if you are working on an
initiative of
> > your own, you should consider these ten activities particularly
> > carefully (not to mention the other 100)."
> >
> > p. 48: "The chart below shows how many of the nine
organizations we
> > studied have adopted various activities. Though you can use
this as a
> > rough ?weighting? of activities by prevalence, a software
security
> > initiative plan is best approached through goals and objectives."
> >
> > Your words (...BSIMM fails...) imply (to me) that you posit
> > organizations attempting to use the collected wisdom in BSIMM
will,
> > inexplicably, look at it and say "Okay, we have to do all 110 of
> > these things exactly as written, so let's get started"
without regard
> > to their local need. This is as opposed to, say, looking at
it and
> > thinking "Here's what nine companies have spent dozens of
> > person-decades and millions of dollars learning about what works;
> > let's see what we can glean from that." Uhmmmm, okay.
> >
> > Yes, previous models exist. Although it may have come up in
> > conversation, we did not ask any of the nine something like "What
> > model did you start with back in the beginning?" because it
simply
> > isn't relevant to what we're trying to accomplish
(documenting what
> > successful organizations are doing), just as "could" and "should"
> > aren't relevant. We asked "What *are* you doing now?" and
documented
> > it so others could learn from it.
> >
> > --Sammy.
> >
> > -----Original Message----- From: Pravir Chandra
> > [mailto:chandra at list.org] Sent: Wednesday, March 11, 2009
4:00 AM To:
> > Sammy Migues; sc-l-bounces at securecoding.org;
sc-l at securecoding.org
> > Subject: Re: [SC-L] Positive impact of an SSG
> >
> > Yes, I don't think its exclusive to your BSIMM interviews
that you
> > found when people put controls into place, they saw improvement.
> > That's what I (and I'm sure many other consultanting firms)
have been
> > doing for years based upon previous models (CLASP, MS SDL, etc.).
> > Nothing to do with BSIMM per se (actually, most of what DTCC
started
> > doing was based on CLASP), just that they added controls
'early into
> > software development lifecycle' and saw benefit, which isn't
> > surprising.
> >
> > That being said, the important part we're missing as 'software
> > security guys' isn't the specification of all the possible things
> > that an organization *could* do, but rather what a given
organization
> > *should* do based on good business decisions around risk
management.
> > And that's the crux of what BSIMM fails to do. By basing the
current
> > maturity model on the collected practices of 9 massive firms that
> > spend the most on that problem, anyone (aside from firms in a
similar
> > situation to your 9) that attempts to apply it to their
organization
> > effectively throws risk management decisions out the window and
> > commits to a much more costly solution than they could have
created
> > based on the knowledge of their own business needs since all the
> > practices are based solely on the behaviors of the select few
firms
> > you interviewed. I'm not discounting the validity of the
empirical
> > data, I'm just positting that it isn't scientifically valid for
> > solving the problem at hand.
> >
> > I'm interested to hear what you learn when you get to the
small and
> > medium sized businesses as well as firms using agile development
> > models (something I particularly considered and accounted for
with
> > SAMM).
> >
> > Regardless of whether we agree on the percentage of orgs for
which a
> > dedicated SSG isn't cost effective, I'm sure we can agree that
> > affording 'someone in charge of success' doesn't equate to a
> > dedicated SSG. There's a myriad of ways that can be
accomplished in
> > any organizational structure.
> >
> > Thanks!
> >
> > p.
> >
> > ~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~ ~~~~~ ~~~ ~~ ~
Pravir
> > Chandra chandra<at>list<dot>org PGP: CE60
> > 0E10 9207 7290 06EB 5107 4032 63FC 338E 16E4 ~ ~~ ~~~ ~~~~~
> > ~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
> >
> > -----Original Message----- From: Sammy Migues
<SMigues at cigital.com>
> >
> > Date: Tue, 10 Mar 2009 23:15:39 To:
> > sc-l at securecoding.org<sc-l <sc-l at securecoding.org<sc-l>
> <sc-l at securecoding.org<sc-l>
<sc-l at securecoding.org<sc-l>>@securecoding.org> Subject: Re: [SC-L]
> > Positive impact of an SSG
> >
> >
> > Hi Pravir,
> >
> > Yes, I agree completely: the data gathered in the BSIMM
interviews
> > seem to indicate that "the controls over all" led to what the
> > interviewees saw as improvements in their capability to produce
> > secure software.
> >
> > In the nine companies interviewed, those controls (BSIMM
activities,
> > I think) sprang from well established SSGs -- that is, a specific
> > person or persons with the responsibility for ensuring lots (110,
> > collectively) of activities actually get done.
> >
> > The BSIMM data to date from specific large organizations indicate
> > that a little under 100:1 is the average ratio for dev/QA to SSG
> > size. It'll be interesting to see how this changes when we get to
> > interviewing smaller organizations and we see if and how they're
> > actually getting it done.
> >
> > Personally, I don't believe I agree with your guess that 95% of
> > organizations building code can't afford an SSG. I believe every
> > organization that wants to succeed can afford to have someone in
> > charge of success, but that's just my opinion and isn't
relevant to
> > BSIMM.
> >
> > Cheers,
> >
> > --Sammy.
> >
> >
> > -----Original Message----- From: Pravir Chandra
> > [mailto:chandra at list.org] Sent: Tuesday, March 10, 2009 6:31
PM To:
> > Sammy Migues Cc: sc-l at securecoding.org Subject: Re: [SC-L]
Positive
> > impact of an SSG
> >
> > Hey Sammy.
> >
> > How does that pertain to a software security group (SSG) per
se? The
> > details below seem to indicate that it was the controls over
all that
> > lead to the positive impact.
> >
> > My main point is that supporting an SSG isn't cost effective
for 95%
> > of the organizations out there that are building code. That's
why in
> > SAMM, we didn't mandate the structure of the organization and
instead
> > concentrated on the functions fulfilled by security guys
(regardless
> > of their placement in the org).
> >
> > p.
> >
> > On Tue, Mar 10, 2009 at 7:48 AM, Sammy Migues
<SMigues at cigital.com>
> > wrote:
> >> Hi all,
> >>
> >> I've received some private questions about the 110 activities in
> >> BSIMM (bsi-mm.com). Since we built the model directly from
the data
> >> gathered, each activity is actually being done in one of the
nine
> >> organizations interviewed. The question is whether there's any
> >> evidence the activities are actually effective as opposed to
simply
> >> being done.
> >>
> >> Since we can't publish any private data, I'd like to point
folks at
> >> this recent article in Information Security Magazine. Jim Routh,
> >> CISO of DTCC (one of the nine organizations interviewed), is
quoted
> >> as follows relative to the impact of software security group
> >> activities:
> >>
> >>
> http://searchsecurity.techtarget.com/magazineFeature/0,296894,sid14_gci1346974,00.html
> >>
> >>
> >> "One of Routh's big wins is inserting security controls
early into
> >> software development lifecycle at the DTCC. Vulnerabilities are
> >> weeded out well before they appear in functional code that
ends up
> >> in production and that has resulted in close to $2 million in
> >> productivity gains on a base of $150 million spend for
development,
> >> Routh says.
> >>
> >> "Those gains are exclusively the result of having mature and
> >> effective controls within our system and software development
> >> lifecycle," Routh says. This is a three-year-old initiative that
> >> educates and certifies developers in all DTCC environments in
> >> security. Developers are also provided with the necessary
> >> code-scanning tools and consulting and services help to keep
> >> production code close to pristine."
> >>
> >> --Sammy.
> >>
> >> Sammy Migues Principal, Technology 703.404.5830 -
> >> http://www.cigital.com Software confidence. Achieved.
> >> smigues at cigital.com
> >>
> >>
> >>
> >> _______________________________________________ Secure Coding
> >> mailing list (SC-L) SC-L at securecoding.org List information,
> >> subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List
> >> charter available at -
http://www.securecoding.org/list/charter.php
> >> SC-L is hosted and moderated by KRvW Associates, LLC
> >> (http://www.KRvW.com) as a free, non-commercial service to the
> >> software security community.
> >> _______________________________________________
> >>
> >
> >
> >
>
> --
> Benjamin Tomhave, MS, CISSP
> falcon at secureconsulting.net
> LI: http://www.linkedin.com/in/btomhave
> Blog: http://www.secureconsulting.net/
> Photos: http://photos.secureconsulting.net/
> Web: http://falcon.secureconsulting.net/
>
> [ Random Quote: ]
> "Don't you wish there were a knob on the TV to turn up the
intelligence?
> There's one marked 'Brightness,' but it doesn't work."
> Gallagher
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc -
> http://krvw.com/mailman/listinfo/sc-l
> List charter available at -
http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC
> (http://www.KRvW.com)
> as a free, non-commercial service to the software security
community.
> _______________________________________________
>
--
Benjamin Tomhave, MS, CISSP
falcon at secureconsulting.net
LI: http://www.linkedin.com/in/btomhave
Blog: http://www.secureconsulting.net/
Photos: http://photos.secureconsulting.net/
Web: http://falcon.secureconsulting.net/
[ Random Quote: ]
"Why don't they make the whole plane out of that black box stuff."
Steven Wright
------------------------------------------------------------------------
_______________________________________________
Secure Coding mailing list (SC-L) SC-L at securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
_______________________________________________
-- Benjamin Tomhave, MS, CISSP falcon at secureconsulting.net LI: http://www.linkedin.com/in/btomhave Blog: http://www.secureconsulting.net/ Photos: http://photos.secureconsulting.net/ Web: http://falcon.secureconsulting.net/ [ Random Quote: ] Osborn's Law: "Variables won't; constants aren't." http://globalnerdy.com/2007/07/18/laws-of-software-development/
Current thread:
- Positive impact of an SSG Sammy Migues (Mar 10)
- Positive impact of an SSG Pravir Chandra (Mar 10)
- Positive impact of an SSG Sammy Migues (Mar 10)
- Positive impact of an SSG Pravir Chandra (Mar 11)
- Positive impact of an SSG Sammy Migues (Mar 11)
- Positive impact of an SSG Benjamin Tomhave (Mar 11)
- Positive impact of an SSG Brian Chess (Mar 11)
- Positive impact of an SSG Pravir Chandra (Mar 11)
- Positive impact of an SSG Benjamin Tomhave (Mar 11)
- Positive impact of an SSG Brian Chess (Mar 11)
- Positive impact of an SSG Benjamin Tomhave (Mar 11)
- Positive impact of an SSG Sammy Migues (Mar 10)
- Positive impact of an SSG Pravir Chandra (Mar 10)
- Positive impact of an SSG Pravir Chandra (Mar 11)