- Case study
- Open Access
Applying statistical process control in small sized evolutionary projects: results and lessons learned in the implementation of CMMI-DEV maturity level 5 in Synapsis Brazil
Journal of Software Engineering Research and Development volume 2, Article number: 2 (2014)
This paper presents the roadmap adopted by Synapsis Brazil to implement the process areas of CMMI-DEV Maturity Level 5 in small sized evolutionary projects. We also discuss the results of applying statistical process control in more than 20 projects, and the lessons learned in conducting successfully a SCAMPI A assessment of CMMI-DEV Maturity Level 5.
Process maturity models like CMMI-DEV and Brazilian model MPS.BR are adopted by organizations as a reference for software process improvement. Many benefits are observed by organizations that adopted such models, for instance, increase in software development team productivity, reduction of projects schedule and costs, and increase in software product quality (Travassos and Kalinowski 2012).
Nonetheless, CMMI-DEV and MPS.BR assessments concentrate on the lower maturity levels, making more difficult to obtain benefits with implementation of higher maturity level best practices. According to (SOFTEX 2013), less than 2% (8 out of 440) of all MPS.BR assessments conducted in Brazil were on the higher maturity level (level A), and 88% of the assessments concentrated on the first two maturity levels (G and F). According to (SEI 2012), only 6% (11 out of 174) of the official valid assessments of SCAMPI A of CMMI-DEV model in Brazil were on the highest maturity level (level 5), while 50% of these assessments concentrated on the lowest two maturity levels (1 and 2).
Many factors difficult the successful execution of software process improvement initiatives. According to (Montoni and Rocha 2011), many of these factors are related to software organization members (for instance, easy to accept changes and higher management support), and also to factors related to consultancy organization members (for instance, knowledge, experience and skills of members of the consultancy) and factors related to organizational strategy and policies (for instance, inexistence of conflict interest in the software process implementation initiative), among others. Therefore, it is likely that many of these factors may also difficult the conduction of software process initiatives in higher maturity levels, like CMMI level 5 and MPS level A.
This paper presents the roadmap adopted by Synapsis Brazil to overcome critical success barriers of the implementation of process areas of level 5 of CMMI-DEV in small sized evolutionary projects. The paper also discusses the results of the application of statistical process control in more than 20 projects, and the lessons learned in the successful conduction of a SCAMPI A assessment of CMMI-DEV maturity level 5 after only 12 months of process implementation with the support of ProMove, a consultancy specialized in software process improvement.
In the CMMI maturity levels 2 and 3 and MPS.BR maturity levels G to C the control of the software process is reactive. In higher maturity levels (CMMI levels 4 and 5, for instance) software process control shall be proactive, based on its performance (Luo et al. 2008; Florac et al. 2000). In this context, traditional software measurement methods and techniques such as comparing planned versus actual data are not sufficient to predict process performance.
One of the mayor difficulties to achieve higher maturity levels in software development is the lack of adequacy of the collected measures and measurement database available in software organizations to apply statistical process control, and the difficulty to access project data (Kitchenham et al. 2006). Therefore, organizations seeking to achieve high maturity in their process shall measure software process efficiently since the early stages of a process improvement program.
Some organizations get frustrated with the assessment of process performance by using statistical process control mainly because they start using these powerful techniques to measure large processes, composed of a great variety of subprocesses (Florac et al. 2000). Statistical process control do not need to be applied in all processes or process components that compose the organization software development process, but it shall be applied only to critical process and not all statistical process control techniques are applicable to all software processes (Sargut and Demirors 2006).
Statistical process control focus in controlling key processes and process components. Moreover, in order to be effective certain requirements must be satisfied, such as reasonable amount of observations, existence of controlled processes and performance objectives aligned to business strategy, definition of measures associated to activities that produce tangible products, and product quality characteristics defined by the clients should be used to select and define measures. Moreover, a process or process component should be easy to control, i.e., should be defined appropriately and of short duration (Sargut and Demirors 2006; Weller et al. 2008).
Synapsis Brazil is one of the most relevant Latin America IT services and technology integration organization, with a wide portfolio of IT services that include IT outsourcing, cloud computing services, IT consultancy and SAP, software factory, technical and business solutions, smart grid solutions and development/implementation of customized solutions. Synapsis Brazil is also leader in specific segments as utilities, oil and gas, and Brazilian govern sectors. The company is also aggressively expanding in other vertical industries with significant presence in Argentina, Brazil, Chile, Colombia and Peru.
In order to demonstrate excellence in software development, Synapsis Brazil invested highly in the last years in the improvement of its processes. The organization was successfully assessed in CMMI maturity levels 2 and 3 in August 2006 and July 2009, respectively. In December 2011, Synapsis Brazil higher management established as a process objective to achieve in December 2012, the official assessment of CMMI maturity level 5. This objective was considered ambitious since the average of the organizations take 2 years to move from CMMI level 3 to 5 (SEI – Software Engineering Institute 2012) (double of the time established by Synapsis Brazil higher management). Even though the objective was ambitious, Synapsis Brazil conducted successfully an official SCAMPI A appraisal on CMMI maturity level 5 in December 2012, after 12 month of process improvement implementation.
One of the factors that most contributed to the success of the initiative was the higher management support. Since the beginning of the implementation, Synapsis Brazil higher management dedicated sufficient human and financial resources to successfully implement CMMI level 5 process areas. Moreover, they participated actively of all meetings to analyze the situation of organization process performance. The support of higher management was also essential in the establishment of quality and process performance objectives. After many meeting with the process group, it was established as performance objective that the project real productivity should be equal or higher than 90% of the expected productivity, whilst the quality objective were that the client acceptance test approval should be equal or higher than 95%. Table 1 presents the evolution of Synapsis Brazil processes to achieve these objectives.
In order to achieve quality and performance objectives established by Synapsis Brazil, first we conducted an analysis of the organizational process baselines that were adherent to CMMI level 3 practices adopted in the projects in the period of May to December 2011 (baselines 14 to 19 of Table 1). The analyses demonstrated that the quality and performance objectives were not being achieved and that several process improvements were necessary to achieve the goal. One of the main improvements conducted were the process optimization. This optimization was executed by automating process activities, and by eliminating redundant standard process activities that do not aggregate direct value to the software product. As a result, we obtained an optimized process with a minimum set of activities easy to understand, plan, monitor, measure and execute. After implementing the process optimization, 77% of the process activities were eliminated (104 out of 135 activities) without losing adherence to CMMI level 3 practices. This reduction were critical to allow the adoption of CMMI level 5 practices not only in small sized evolutionary projects, but also in large software development projects.
In the period of January to July 2012, were published baselines with not only the purpose of piloting incremental improvements in processes and templates, but also aiming to implement innovative improvements such as adaptation of the standard process to small sized evolutionary projects (less than 200 man-hour), and adoption of code inspection and peer review of technical specifications (baselines 20 to 24 of Table 1). The process adaptation strategy to small projects, and the adoption of practices to increase the quality of intermediate software development products (software architecture, test plans, requirement and technical specifications etc.) were considered critical success factors, because help to achieve the organizational quality and performance objectives (assertiveness of estimated productivity and reduction of defects detected by the client). Measurement analysis demonstrated that small projects were easier to be estimated, monitored and controlled, since corrective actions to manage deviations from project plans have more effect on the project than on large projects. Moreover, the anticipation of defects detection helped to improve product quality, and consequently, to reduce the amount of defects detected by the client in the acceptance testes. As a result of the pilot of these improvements, we could observe a tendency in the improvement of quality and productivity.
The next step was the stabilization of the processes by conducting performance analysis of selected subprocesses. The following subprocesses were selected for statistical process control since they are in the critical path of the projects: Specify product requirements, Elaborate technical specification and effort estimates, Code and integrate units, Conduct product tests and Conduct product acceptance tests. Improvements in processes and templates were adopted in new projects (baselines 25 and 26 of Table 1). The specialized knowledge of the consultancy team in high maturity practices, mainly knowledge about statistical process control were considered a critical success factor to conduct performance analysis and to evolve the process baselines. The knowledge acquired by taking specialized courses in the field helped to understand potential problems that affect process performance, and to suggest incremental and innovative improvements that could help to achieve organizational quality and performance objectives. Nevertheless, performance analysis of baseline data demonstrated that although the performance objective were achieved, it was not possible to achieve the quality objective, since many defects continued to be detected by the client in the acceptance tests.
The final step was to pilot innovative improvements like the adoption of peer programming and test coverage analysis in the projects (baseline 27). The performance analysis of the processes executed in the pilot projects confirmed the improvement in the quality and process performance. Consequently, we could successfully achieve the objectives established by the organization. The easy of process change acceptance and the tool support for statistical process control were considered critical success factors in this phase. During the conduction of the pilots, many improvements were implemented in the projects, including during projects execution, requiring some rework by the project managers and the development team. All project members were open to process changes, helping the organization to achieve its objectives. The adopted tools for statistical process control such as MiniTab™ and Oracle Crystall Ball™ reduced considerably the effort to perform data analysis; helping to rapidly verify if subprocesses attributes relations were statistically relevant.
Discussion and evaluation
In the last section, the knowledge about statistical process control, and the use of support tools to statistical process control were considered critical success factors in the implementation of high maturity practices in Synapsis Brazil. In order to manage those factors effectively, the process and templates of Synapsis Brazil were adjusted so that the application of statistical process control knowledge and techniques could be easily adopted by project managers during quantitative management of small sized evolutionary projects. This section presents some of the results obtained with the implementation of these improvements.
The lack of an adequate measurement repository in the market to support statistical process control was a problem for the implementation of statistical software process control. In order to minimize this problem, Synapsis Brazil implemented a specific measurement repository to support adoption of statistical software process control techniques. The adopted strategy in the implementation of the repository and in loading the necessary data was the following: (i) Defect information originated in Excel spreadsheets in which we implemented specific entries to facilitate collection of measures and to load data in the measurement repository; (ii) Information of estimated man-power effort, activities, process, size, complexity, productivity are loaded in the planning and monitoring tool used by project managers from size and effort data extracted of the estimation tool; (iii) Information of actual effort are loaded in the planning and monitoring tool used by project managers from data extracted of the Jira™ workflow tool; and (iv) Information about activities executed, estimated man-power effort, actual effort, activities schedule, productivity, product size, team, profile, technology etc., are loaded from measurement data extracted from the planning and monitoring tool. The processing of data stored in the repository, and two tools supported the visualization and generation of control graphics: Oracle Crystal Ball™ e Minitab™.
The lack of adequate tools to support quantitative management motivated the elaboration of a spreadsheet to support project managers in storing baseline and prediction models data. Figure 1 presents an example of the support tool to quantitative management. The tool presented in Figure 1 supports two statistical process control mechanisms. The top of the figure presents a control graphic automatically generated from project data. The bottom part of Figure 1 presents the probability analysis of not achieving the project quality and performance objectives based on the prediction models developed. By using the spreadsheet, statistical knowledge becomes transparent, thus facilitating its use and diminishing resistance by the project team.
It was observed that one of the most complex activities to be executed involved the selection of which subprocesses attributes should be controlled to address the established quality and performance objectives. The analysis of existence measurement data in the repository indicated that some of the characteristics of the subprocesses have greater influence in achieving the quality and performance indexes established by the organization. The combination of information about the characteristics of the subprocesses and other project information, such as characteristics of the team allocated in the project, allowed the project managers to verify the probability of achieving the quality and performance objectives every time the data were updated, thus providing the means to manage action plans to minimize associated risks of not achieving the objectives. This mechanism of subprocesses and its attributes selection not only simplified the implementation of a complex high maturity practice, but also facilitated the understanding of the process selection concept by the project managers.
Synapsis Brazil process group identified that the Minitab™ tool to support data regression analysis helps not only in the early stages of performance analysis, for instance, to verify correlation of subprocesses attributes selected for statistical process control, but also supports the calculation of prediction models adopted in the quantitative management of Synapsis Brazil projects. For instance, the following performance model was developed by the application of regression analysis technique supported by MiniTab™: Percentage of Approved = 0,818 + 0,0717 * Amount of Criteria by Function Point. This model indicates that exists a positive correlation among the amount of criteria by function point in the product tests and the percentage of test cases approved in the product acceptance tests by the client. One of the problems of application of regression analysis is the identification of outliers, i.e., specific data points that do not represent a common behavior in the project. MiniTab™ helped to automatically identify the outliers, thus facilitating the elimination of anomalies in the data that hindered the development of trustful models.
We also noticed that the sensitivity analysis technique of Monte Carlo simulation using Oracle Crystal Ball™ tool helped to identify critical subprocesses that affected the achievement of quality and process performance objectives of Synapsis Brazil, such as Product Tests subprocesses. After the implementation of incremental and innovative improvements in those subprocesses, we could verify a significant change in the processes performance, helping the organization to achieve its objectives. One of the problems in the application of Monte Carlo simulation was the lack of past project data necessary for conduction of simulations. As the understanding of critical subprocesses and its relationships increase, new measures are necessary for performing the analysis. Nevertheless, many times these measures are not found in the repository. This problem forced the elimination of several projects in the early stages of baselines data analysis due to lack of data that could be used in the simulation analysis.
The implementation of software process improvement is a complex activity that requires significant amount of human and financial resources. Moreover, the return of investment is observed in a long-term period. Nevertheless, software process improvement is likely to pay-off when the improvements are adopted across the organization and software development maturity is consistently observed during projects execution. We could observe some significant results after successfully conducting software process improvement initiatives in Synapsis Brazil. Table 2 presents some of the direct results in defect density, productivity and schedule observed after implement CMMI maturity levels 2, 3 and 5.
Table 2 shows that there was a significant reduction of defect density from level 2 to level 3 (42%). This was considered a good result since the focus of CMMI level 3 process areas is product engineering and verification and validation practices. Although the reduction of defect density from CMMI level 3 to level 5 were smaller (16%), Synapsis Brazil higher management was satisfied with the result since it was possible to achieve the business objectives concerning product quality.
With respect to productivity, we could observe a continuous improvement from level 2 to 3 (19%) and from level 3 to 5 (15%) in Table 2. One of the causes of this result was the process optimization performed during CMMI level 3 and 5 implementation thus reducing the overall effort to develop software.
The best results were observed in the percentage of delayed projects in Table 2. This percentage dropped from 45% of delayed projects to 20% after implementing CMMI level 3 and continued to drop to 4% of delayed projects after implementing CMMI level 5. The reduction of project size can be one of the causes of this significant improvement. Moreover, the implementation of quantitative project management practices and statistical process control techniques helped to prevent future process performance and to take more effective corrective actions during projects execution.
The Synapsis Brazil process group observed as a lesson learned the fact that the process implementation aiming high maturity levels do not represent greater effort related to CMMI level 3 implementation. The project team is not significantly affected by the implementation, since the new process areas to be implemented in the standard processes do not affect directly the team. The measurement group is the most affected group by high maturity implementation since they have to assist project managers, some members of the project team and the higher management with control information so that improvement actions can be planned and executed in time to correct deviations before concluding projects activities. Another issue that the process group considered as a differential was the facility that the project manager had to initiative the project planning concerning subprocesses definition that compose the adapted process to the project. The activity of subprocesses selection is fundamental to assess achievement of quality and performance objectives with the composition of the project process. This activity was facilitated by the automated support in the selection of subprocesses based on project characteristics and organizational performance baselines.
Project managers observed that the available tools and process and quality groups were the main structure to support quantitative management, providing the mechanisms for planning and controlling projects activities, i.e., those groups not only supported project managers in the effective planning, but also supported the project team to solve problems during projects execution. Risks planning and monitoring helped to increase the understanding of factors that may jeopardize projects execution concerning achievement of quality and performance objectives. A continuous monitoring minimizes the occurrence of problems during projects execution, since it is fundamental to be aware of the situation of projects products, team and client commitments and accomplishment of established milestones and budget. The project managers also noted that the process efficacy is important but the team involvement and commitment to work have also great influence in projects success. People continue to be the essential part of the process and project managers should always keep that in mind. In this work, the great differential was the use of statistical data. Project managers noted that using these data in an adequate manner increases the chances of predicting future process performance aiming to facilitate achievement of project objectives.
The consultancy observed that the lack of data jeopardizes the assessment of process performance. A way to overcome this problem is to focus the analysis in evolutionary projects of small size, because they allow rapid data generation. The development of initial prediction models with few independent variables (one to two at most), also helps to obtain more trustful models when few data is available.
This paper presented the experience of Synapsis Brazil in the evolution of its processes to achieve high maturity. We presented the main critical success factors, such as easy to accept changes by project teams, support and commitment of higher management, adequate supporting tools for statistical process control, incremental and innovative improvements and the specialized knowledge of the consultancy in statistical process control. The paper also discussed the results of adopting statistical process control in small sized evolutionary projects. Lessons learned by the teams were also discussed.
Considering that the purpose of CMMI level 5 is continuous optimization, the next steps for Synapsis Brazil process improvement is to analyze performance of the actual baseline aiming to identify the subprocesses that affect the achievement of quality process performance objectives reviewed by the higher management aiming a more competitive position both in the national and international software market.
Florac WA, Carleton AD, Barnard JR: Statistical process control: analyzing a space shuttle onboard software process. IEEE Software 17:97–106. n 4, July/August. CA, USA: IEEE Computer Society Press Los Alamitos; 2000.
Kitchenham B, Kutay C, Jeffery R, Connaughton C: Lessons learnt from the analysis of large-scale corporate databases. ICSE '06 Proceedings of the 28th international conference on Software engineering. NY, USA: ACM New York; 2006:439–444.
Luo W, Wang M, Zhou B, Liu P: Research on CMMI-based project management environment, 2008 International Conference on Wireless Communications, Networking and Mobile Computing. Dalian, China: WiCOM 2008; 2008.
Montoni MA, Rocha AR: Using grounded theory to acquire knowledge about critical success factors for conducting software process improvement implementation initiatives. Int J Knowl Manag (IJKM) 2011, 7(3):43–60.
Sargut KU, Demirors O: Utilization of statistical process control (SPC) in emergent software organizations: pitfalls and suggestions. Software Quality Journal. Kluwer Academic Publishers 2006, 14(2):135–157.
SEI – Software Engineering Institute: CMMI for SCAMPI class a appraisal results 2012 Mid-year. 2012.http://cmmiinstitute.com/assets/presentations/2012%20SepCMMI.pdf Update, , accessed in April/2013
SOFTEX: Total number of organizations with MPS Assessment (valid or not valid) (in Portuguese). 2013.http://www.softex.br/mpsbr/_avaliacoes/avaliacoes_mpsbr_total.pdf , accessed in April/2013
Travassos GH, Kalinowski M: iMPS 2011 Performance Results of Organizations that Adopted the MPS Model from 2008 to 2011 (in Portuguese). Campinas/SP: SOFTEX; 2012:36. , accessed in April/2013 (2012) http://www.softex.br/wp-content/uploads/2013/08/iMPS-2011-Resultados-de-Desempenho-das-Empresas-que-Adotaram-o-Modelo-MPS-de-2008-a-2011.pdf , accessed in April/2013 (2012)
Weller E, Card D, Curtis B, Raczynski B: Point/counterpoint applying SPC to software development: where and Why, software. IEEE Software 25(3):48–51. May-June. CA, USA: IEEE Computer Society Press Los Alamitos; 2008.
The authors declare that they have no competing interests.
MM identified common problems of similar cases, elicited some critical barriers for conducting the reported experience and drafted the manuscript. CS documented the case results and lessons learned. Both authors read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.