Improving Software Reliability Using In-process Metrics

Industrial strength software development enforces numerous processes to deliver the product ontime. To improve software quality, including reliability, processes are often modified. However, it is often difficult to understand the impact of any one process change or addition on the overall software reliability, since it can frequently take many months to get sufficient feedback from the field. If software measurements made early in the lifecycle can be found to predict ultimate release quality, we can then effectively isolate the influence of specific processes/practices and tune them appropriately to improve the resulting quality.
 
This workshop addresses how to:
  • gather, scrub, automate, and work with in-process metric data
  • build models that enable early predictions
  • identify specific in-process metrics that are useful as predictors.
These topics will be covered:
  1. Data for models. Discuss the difficulties in gathering and scrubbing adequate data for model variables. Emerging software development platforms such as Microsoft Visual Studio Team System (VSTS) and IBM Team Concert (aka IBM Jazz) provide mechanisms to collect various in-process events, in addition to collecting basic structural information about the software. This section focuses on automation methods, use of dashboard data, and metric goaling strategies based on model results. We will examine case studies, emphasizing specific data-related issues and resolution approaches.
  2. Empirical and deterministic model development. Linear and nonlinear regression, CART, neural/Bayesian network, bugflow, and other model classes will be discussed, emphasizing the plusses and minuses of each. Case studies will be presented and discussed. Predictions of customer-found defects, total post-release defects, defect escape volumes, customer satisfaction scores, and security vulnerabilities will be described. Analysis techniques will be addressed, including constructing confidence intervals, goodness of fit statistics, and cost models.
  3. Using models for diagnosis and prognosis. Early predictive modeling is improved by capturing near-real time events during software development and applying statistical techniques on the events and other software artifacts. The underlying premise is that this approach provides earlier warning, more accurate diagnosis of problems, and faster remedies than models that rely on off-line, downstream events, thereby improving overall reliability of the software process and products.   Predictive models that use data from defect and trouble ticket databases will also be described and discussed. We will cover case-studies, diagnosis and predictive metrics, end-user enablement, and practical process improvement.
Workshop Agenda
Morning:
  • Keynote [30 min.]
  • Invited Paper #1: Data for models – where and how to automate [20 min., plus 10 min. Q&A]
  • Invited Paper #2: Building models - choosing the right framework [20 min., plus 10 min. Q&A]
  • Accenture Case Studies (covering data, model building, and predictions) [30 min., plus 10 min. Q&A]
  • Summary of morning events, identify afternoon discussion topics, overview of afternoon session [30 min.].
Afternoon:
  • Overview of afternoon activities [10 min.]
  • Invited Paper #3: Prediction model – practical example [20 min., plus 10 min. Q&A]
  • Cisco Case Studies (covering data, model building, and predictions) [30 min., plus 10 min. Q&A]
  • Breakout groups – 1. data management, 2. modeling and simulation [40 min.]
  • Discussions of 1 topic from each of the 2 groups [40 min.]
  • Wrapup [10 min.].
Call for papers.  Three short papers will be presented at this workshop: 
 
a. Invited paper #1: Data for models – where and how to automate data collection.  Practical experience in instrumenting code and using other means to gather, store, analyze, and use in-process measurement data and metrics.
b. Invited paper #2: Building models – choosing the right framework.  Method used to choose the right modeling approach; limitations and strengths of various options; tools available.
c. Invited paper #3: Prediction model – practical example.  Modeling and simulation results for reliability prediction model (or another similar type of model, such as software security, customer experience, etc.). 
 
You are asked to submit a one-page abstract - deadline TBD.  Slides will be used for the workshop – a maximum of 12 slides, please, for 20 min. presentations, followed by 10 min. of Q&A.  Please send submissions to:  Pete Rotella, protella (at) cisco.com.