Home page logo

securecoding logo Secure Coding mailing list archives

Re: BSIMM-V Article in Application Development Times
From: John Steven <jsteven () cigital com>
Date: Wed, 8 Jan 2014 13:52:43 -0500

Christian, (Stephen)

I’ll confess I’ve only skimmed the discussion but it looks productive. The questions posed are good ones. I’ll try to 
provide a few clarifications from “inside” the BSIMM study that may be helpful in pushing the discussion along:

1) Survey structure/technique attributes BSIMM activities by first seeking
    certain confirmations. If interview-based confirmation doesn't provide
    confidence, the subject is asked for documentation. 

2) The BSIMM document indicates who is interviewed, but it’s not an 
    exhaustive list. Where confirmation is necessary Dev/Config 
    Management, architects, and others make the list. 

3) Surveying claims to (and in practice) stops short of concrete attestation
    across the board. 

4) BSIMM survey targets have included organizations, business units, and 
    in rare cases, smaller scopes.

5) At the organization-level, survey confirmation includes facilities to 
    differentiate “one group does it”,  “this is done by most”, and “the
    organization governs mandatory activity conduct”.

6) An organization does/should not get credit for “one group does it”. 

7) More qualified BSIMM interviewers exist than Sammy et al. More are 
    minted as the study grows in size. There isn’t a written certification and
   a pin, but there is an involved apprenticeship. And, Sammy runs cross-
   checking of the grading process to make sure that interviewers remain
   convergent in grading criteria. 

Addressing another question raised by the email chain below: just because the organization does the activity—as a 
rule—doesn’t mean that every team does it. Non-complaince may be a reason. Another (better) reason may be that the 
organization takes a “risk-based approach” to the activity. In other words, an organization may choose to do more 
mature architecture analysis activities on only a subset of applications—those that are higher risk.  This is what 
BSIMM activities Strategy and Metrics (SM) Level 3 are about.

Hopefully that helps a bit. 
John Steven                 
iCTO, Cigital
+1,703-727-4034   |  @M1splacedsoul

On Jan 7, 2014, at 8:07 PM, Christian Heinrich <christian.heinrich () cmlh id au> wrote:


On Sat, Jan 4, 2014 at 8:12 PM, Stephen de Vries
<stephen () continuumsecurity net> wrote:
Leaving the definition of agile aside for the moment, doesn’t the fact that the BSIMM measures
organisation wide activities but not individual dev teams mean that we could be drawing inaccurate
conclusions from the data?  E.g.  if an organisation says it is doing Arch reviews, code reviews and
sec testing, it doesn’t necessarily mean that every team is doing all of those activities, so it may give
the BSIMM reader a false impression of the use of those activities in the real world.

In addition to knowing which activities are practiced organisation wide, it would also be valuable to
know which activities work well on a per-team or per-project basis.

My reading of the "Roles" section of BSIMM-V.pdf is that the people
interviewed for the BSIMM sample are:
1. Executive Leadership (or CISO, VP of Risk, CSO, etc)
2. Everyone else within the Software Security Group (SSG)

What you are asking to be included is what is referred to as the
"Satellite" within BSIMM-V.pdf and I believe this may also require the
inclusion of http://cmmiinstitute.com/cmmi-solutions/cmmi-for-development/
too (why not :) ).

The issue with this is that it would invalidate the statistics from
the prior five BSIMM releases due to the inclusion of new questions
and in additional these new statistics were not gathered over time
either hence the improvements measured over time within BSIMM would be
invalid too due tot he new dataset.

Furthermore, Gary, Sammy and Brian have limited time to interview all
67 BSIMM participating firms.

However, I would be interested to know the "BSIMM Advisory Board" i.e.
http://bsimm.com/community/ view on this is and if it would be
possible to undertake this additional sampling within their own BSIMM
participating firm to determine if there is additional value would be
gained for BSIMM?  However, I suspect that an objective measurement
would be too hard to quantify due to internal politics of each BSIMM
participating firm but I could be wrong.

Secure Coding mailing list (SC-L) SC-L () securecoding org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]