- Open Access
Improvement of IT service processes: a study of critical success factors
Journal of Software Engineering Research and Developmentvolume 2, Article number: 4 (2014)
Maturity models and Information Technology (IT) service management models guide the definition and improvement of service management processes. Known approaches include ITIL, COBIT, ISO/IEC 20000, CMMI-SVC, and MR-MPS-SV. The implementation of these models results in benefits such as: increased user and customer satisfaction with IT services; financial savings due to less rework and less time used, and improved resource management and usage; improved decision making and optimized risk; and better alignment based on the business focus. However, some organizations find it difficult to use the models. This paper presents a study in which we identified critical factors for success and failure of the improvement of IT service processes. By doing a systematic mapping study and by snowballing, we were able to identify factors such as: project implementation strategy; support, commitment, and involvement; processes; and internal and external resources. Also, we analyzed our results using grounded theory procedures in order to facilitate their understanding.
A service is a way of delivering value to customers, while facilitating the achievement of the results they want to obtain without having to deal with unnecessary and risky costs (SEI,2010; Cartlidge et al.2007). An Information Technology (IT) service is a set of resources, whether IT or non-IT, perceived by the client as whole, and maintained by an IT provider. Such services aim to satisfy one or more needs of a client and support the strategic goals of his business (Magalhães and Pinheiro,2007).
In order to achieve good performance for the service provider and satisfy the client, it is necessary to develop and improve the practices involved with the services (SEI,2010). In such a context, maturity models and IT service management models have been developed to guide organizations in the definition and improvement of service management processes. For instance, some of the already known and disseminated approaches are: ITIL (von Bon2011), COBIT (ICASA,2012), ISO/IEC 20000 (ISO/IEC,20092010a2010b20112012), CMMI-SVC (SEI,2010), and MR-MPS-SV (SOFTEX,2012).
Several research papers (SEI,2010; Cartlidge et al.2007; and ICASA,2012) have pointed out a series of benefits derived from adopting and implementing service management processes; for example: an increase in satisfaction with the services for both clients and users, higher productivity, improvement in economy by reducing workload and increasing the use of management resources, improvement in decision making, greater alignment between IT and the business goals, and appropriate management of the IT risks. However, there are organizations that suffer difficulties during the implementation of projects related to the improvement of IT service processes, and consequently they do not succeed (Jäntti,2010; Pollard and Cater-Steel,2009; Sharifi et al.2008).
According to Tan et al. (20072009) and Pollard and Cater-Steel (2009), some factors can influence and contribute to the success of an implementation project related to the improvement of IT service processes. These critical success factors are key aspects that must be achieved in order for the business to succeed, and, if not well performed, will make the achievement of the missions and goals unlikely within a business or project (Rockart,19792009). Critical failure factors, or risk factors, can put the improvement of the IT service processes at risk, resulting in implementation failure (Ghayekhloo et al.2009; Wan et al.2008).
Even though Montoni (2010) consolidated critical success factors for the implementation of software processes through a systematic mapping study, we are not aware of a similar study having been performed in the field of IT services. Consequently, the goal of this paper is to identify critical success and failure factors in initiatives for process improvement in the field of IT services. Such information can aid organizations, that are willing to improve their IT services processes, to create implementation strategies with a higher chance of achieving success.
Besides this opening section, this paper presents seven more sections. In Section “Models for improvement of IT service processes” we present the theoretical background for improvement of IT service processes. In Section “Critical factors in improvement of IT service processes” we present a literature review of critical success factors, while in Section “Techniques for reference searching and for data analysis” we provide a discussion on the techniques for reference searching and data analysis used within our study. In Section “Protocol of the systematic mapping and snowballing”, we describe the protocol for performing our study; in Section “Results from the systematic mapping study and snowballing” we provide the results from these studies; in Section “Comparison of critical factors in software process improvement” we present a comparison of the critical factors in software process improvement; and finally, in Section “Conclusions”, we provide final remarks regarding this work.
Models for improvement of IT service processes
The governance of IT services seeks to avoid the occurrence of problems in the delivery and operation of the services provided, in order to guarantee that their quality is perceived by clients and users (Magalhães and Pinheiro,2007). Thus, many models for maturity and management of IT service processes have been proposed, which include approaches such as ITIL (von Bon2011), COBIT (ICASA,2012), ISO/IEC 20000 (ISO/IEC,20092010a2010b20112012), CMMI-SVC (SEI,2010), and MR-MPS-SV (SOFTEX,2012). In the following paragraphs we provide a review of these models.
The Information Technology Infrastructure Library (ITIL) is a framework for better IT service management practices. It is composed of five books that cover the lifecycle of a service, including: business requirements analysis, design, migration to the operational environment, operation and improvement. The Service Strategy book guides service providers to build a strategy for delivering and managing services to support business requirements, while the Service Design book guides the design of appropriate IT services by considering business objectives. The Service Transition book offers guidance to ensure that service releases are implemented successfully into supported environments, and the Service Operation book guides the delivery of services to business users and customers. Finally, the Continuous Service Improvement book offers guidance for achieving improvements in service quality, operational efficiency, and business continuity, as well as to ensure the alignment of the service portfolio and business needs (von Bon2011).
Control Objectives for Information and related Technology (COBIT) is a framework of good practices for effective governance and management over enterprise IT, that is based on five principles (meeting stakeholder’s needs; covering the enterprise end-to-end; applying a single integrated framework; enabling a holistic approach; and separating governance from management) and seven enablers (principles, policies and frameworks; processes; organizational structures; culture, ethics and behavior; information; services, infrastructure and applications; and people, skills and competencies). The COBIT process reference model subdivides the IT-related practices and activities into two main areas: governance and management. The governance domain is: Evaluate, Direct, and Monitor; while the four management domains are: Align, Plan, and Organize; Build, Acquire, and Implement; Deliver, Service, and Support; and Monitor, Evaluate, and Assess (ICASA,2012).
ISO/IEC 20000 is an IT service management standard that is composed of five parts: ISO/IEC 20000–1, ISO/IEC 20000–2, ISO/IEC TR 20000–3, ISO/IEC TR 20000–4, and ISO/IEC TR 20000–5. ISO/IEC 20000–1 specifies the requirements for planning, establishing, implementing, operating, monitoring, reviewing, maintaining, and improving the service management system (ISO/IEC,2011). ISO/IEC 20000–2 and ISO/IEC TR 20000–5 present practices and an example of a management plan to achieve the requirements, respectively (ISO/IEC,20122010b). ISO/IEC TR 20000–3 provides guidelines for defining the scope, applicability, and conformity demonstration of the service management system (ISO/IEC,2009). Finally, ISO/IEC TR 20000–4 describes a process reference model for service management, derived from the requirements described in ISO/IEC 20000–1 (ISO/IEC,2010a).
The Capability Maturity Model for Services (CMMI-SVC) is a maturity model that focuses on companies supplying services and covers required activities for establishing, delivering, and managing services. Thus, there are two incremental paths that can be followed in order to achieve the process improvement: (a) Continuous, in which there is an improvement in an individual process or group of processes that are chosen by the organization; and (b) Staged, in which there is an improvement of a set of related processes already defined by the model. The Continuous representation allows the achievement of capability levels for the chosen process, while the Staged representation allows the organization to achieve evolutionary maturity levels. The capability levels are: (0) incomplete, (1) executed, (2) managed, and (3) defined. The maturity levels are: (1) initial, (2) managed, (3) defined, (4) quantitatively managed, and (5) optimizing (SEI,2010).
The Reference Model for Improvement in Service Processes or MR-MPS-SV (acronym derived from Modelo de Referência de Melhoria de Processos de Serviços in Portuguese) is a Brazilian maturity model for IT service provider companies which was created as part of the MPS.BR Program (Santos et al.2012). The MR-MPS-SV focuses on microenterprises and small to medium-sized companies, and presents seven maturity levels, progressing from G to A: (G) partially managed, (F) managed, (E) partially defined, (D) widely defined, (C) defined, (B) quantitatively managed, and (A) optimizing. The division into seven levels enables the implementation and adequate assessment of microenterprises and small to medium-sized companies. Also, it can increase the visibility of the process improvement results in a shorter amount of time (SOFTEX,2012).
Critical factors in improvement of IT service processes
Some research papers describe perceived critical success or failure factors during the implementation or improvement of IT service processes in provider organizations. For instance, Cater-Steel et al. (2006) present a case study with five organizations that implemented ITIL and, as a result, factors with a positive influence were identified; for example: support from managers to exert pressure towards change, engagement of the affected people, communication of results, reengineering of business processes, marketing campaigns to enhance acceptance and understanding, and the development of the people involved. Besides the above factors, the authors identified factors that had a negative influence; for example, lack of support from managers and resistance to cultural change.
Hochstein et al. (2005) present a case study involving six European companies that finished a project which involved implementing the ITIL. In their study they identified success factors such as employee support, understanding of service oriented processes, dissemination of quick gains, marketing campaigns to enhance acceptance and understanding, management support to exert pressure towards change, and training on a larger scale. They also identified failure factors such as the lack of understanding of the need for new processes.
Pollard and Cater-Steel (2009) compare critical factors that have a positive influence, which were identified in the literature, with factors related to the success of organizations working towards the improvement of IT service processes. Thus, some factors described in the literature were confirmed while others were added; for example, support from senior managers, training of employees, interdepartmental communication and collaboration, and use of external consulting and adequate tools. Wan et al. (2008) analyzed cases, visited companies, and reviewed the literature in order to study factors which have a negative influence on improvement initiatives. They verified 20 critical factors and, by analyzing the causal relationships among them, identified six root risk factors: (a) unclear project goals, (b) inadequate and ineffective communication between the teams, (c) senior leaders do not care about the project, (d) vague business strategy, (e) qualified team members leave the company, and (f) planning and designing of IT infrastructure are unreasonable.
The authors show that there is no consensus regarding which factors are capable of influencing the success of improvement initiatives for IT service processes. Montoni (2010) presented a systematic mapping study that identified and consolidated critical factors that affect initiatives for improvement of software processes. He identified 12 critical factors and, associated with these factors, he identified 25 properties of critical factors for success. Moreover, in his study, he identified 59 categories for findings with a positive and negative influence — these categories characterize either the presence or absence of a property. The factors of “Processes”; “Support, commitment and involvement”; and “Skills of the organization’s members”; cited by Montoni (2010) described almost 50% of the perceived occurrences and, therefore, were considered to be the most critical factors for achieving success in an improvement initiative for software processes. Furthermore, the other set of factors — “Implementation strategy for software process improvement ”, “Resources”, “Awareness of the benefits resulting from the process improvement implementation”, “Organizational structure”, “Conciliation of interests”, and “Policy for recognition of collaboration in the improvement process” — were considered to be less critical due to the medium number of occurrences. Finally, due to the lower number of occurrences, the following factors were considered to be less influencing factors: “Respect for the consultants from the organization’s members”, “Acceptance of changes”, and “Motivation and satisfaction of the organization’s members”. Although the above research provides an insight into success and failure factors in software process improvement initiatives, we are not aware of any similar work about critical success factors in the field of IT services. Therefore, this paper goes one step further by performing a study of this area. In the next section, we describe the techniques we applied in order to perform this study.
Techniques for reference searching and for data analysis
In our study, we used certain techniques to aid our investigation of critical factors. Through a systematic mapping study and snowballing we searched for literature references that contained reports of improvement initiatives in IT service provider organizations. Then, after collecting data from the identified papers, we used grounded theory (GT) procedures to perform analyses. In the following subsections, we will describe each technique.
Systematic mapping study
A systematic literature review (SLR) is a way of searching, evaluating, and interpreting the relevant papers for specific research questions, research areas, or topics of interest (Kitchenham,2004). According to Kitchenham (2004), there are three main stages in the execution of an SLR:
Review planning: In this stage, the researcher creates a review protocol containing the research question, search terms, paper selection criteria, and the data extraction strategy.
Execution of the review: In this stage, the researchers perform a search of the databases, document the executed stages, and select the papers that possess evidence related to the research question. Then, they evaluate the quality of the papers and extract, monitor, and synthesize the data contained within the papers.
Reporting of results: In this stage, the researchers write a report or document containing the results and findings from the SLR.
Systematic mapping studies use the same basic methodology as an SLR; however, they aim to identify all the research related to a specific topic and classify the primary research papers in that specific domain, rather than addressing the specific questions that a conventional SLR addresses (Kitchenham et al.2010). This paper presents the protocol of a systematic mapping study based on the SLR protocol described by Kitchenham (2004).
Snowballing is a method for searching papers in the literature and it involves three steps (Jalali and Wohlin,2012):
Gathering an initial set of papers from key journals and/or conferences related to the research topic;
Analyzing the references of relevant papers identified during step 1, in order to search for new important papers (backward snowballing);
Identifying and analyzing papers that cite the already selected papers (forward snowballing).
The analysis of the references is iterative until there are no new relevant references identified. It is worth noting that there are authors who suggest using snowballing starting from the references of papers that are identified in an SLR, as well as from the papers that are identified in the databases (Jalali and Wohlin,2012). By complementing a review with snowballing, we intend to identify more relevant papers, increase the number of papers about the topic of interest, and complement and enrich the results presented by Diirr and Santos (2013).
Grounded Theory (GT) is a scientific method that uses data collection and analysis procedures to create, elaborate, and validate substantive theories about phenomena and social processes (Bandeira de Mello and Cunha,2003). A theory is a set of concepts that is related by relationship sentences and which, together, constitute a schema that can explain or predict a phenomenon. In other words, theories are derived from data that is systematically unified and analyzed through an investigative process (Strauss and Corbin,1998).
Strauss and Corbin (1998) propose the following types of coding procedures for the GT method:
Open coding: This is an analytical process in which concepts and categories are identified, and their properties and dimensions are discovered in the data. Thus, the data are divided, analyzed, and compared, and their central ideas are represented as concepts that allow the grouping of similar and significant events, incidents, actions, and objects in the data.
Axial coding: This is the process of relating the categories and subcategories to form more precise and complete explanations about the phenomenon. The categories are developed systematically and then related so that they are more in-depth and more structured.
Selective coding: This is the process of integrating and refining the theory by organizing the categories around a central explanatory concept that expresses the essence of the study’s process. Also, possible flaws in the theory are solved and more categories are formulated.
Other authors have used GT to collect and analyze data in the context of papers related to improvement initiatives for software processes (Montoni and Rocha,20102011; Coleman and O’Connor2008). Section VI.B explains how the GT method was applied to this work.
Protocol of the systematic mapping and snowballing
The goal of this systematic mapping study follows the paradigm of the Goal Question Metric (GQM) method (Basili et al.1994) and consists of: Analyzing papers containing reports about improvement initiatives in IT service provider organizations, with the purpose of identifying and analyzing critical success and failure factors, in relation to the influence on the success or failure of improvement initiatives for IT service processes, from the point of view of organizations that implement improvements in IT service processes and the IT practitioners who are involved in such initiatives, and in the context of companies that provide IT services to implement improvements in their IT service processes.
As a result, we defined two research questions to be answered in order to achieve the described goal:
Question 1 Which factors have a positive influence on improvement initiatives for IT service processes?
Question 2 Which factors have a negative influence on improvement initiatives for IT service processes?
Regarding the scope of this research, we adopted the following criteria to select the search sources: (a) the source must be related to the topic being investigated; and (b) the source must either belong to one of the editors listed on the CAPES Journal Web Site (Portal de Periódicos da CAPESa in Portuguese), which is a virtual library that collects and makes available thousands of international scientific publications, or the source must be a symposium, conference, workshop, or similar, that is supported by the Brazilian Computer Society (SBC) — Sociedade Brasileira de Computação in Portuguese — and related to software quality (there is no specific event for IT services supported by SBC), in order to include Brazilian conferences not indexed by international search engines. It is worth noting that this research is restricted to analysis of the papers that were available until the date of its execution (May 2013). Also, we researched papers in English, since it is a language that has been widely adopted in the research area, and Portuguese, since this research was performed in Brazil and we wanted to consider papers published in national conferences.
We selected the Compendex and Scopus digital libraries because, based on other systematic mapping studies we have conducted, we consider their search engines to have good functionality and coverage. Furthermore, we considered that the imminent execution of the snowballing would reduce the risk of not considering other digital libraries such as IEEE and ACM. Also, we selected Brazilian conferences including the Brazilian Symposium of Software Quality (Simpósio Brasileiro de Qualidade de Software or SBQS in Portuguese) and the MPS Annual Workshop (Workshop Anual do MPS or WAMPS in Portuguese).
We used the following search string: (itil OR cobit OR "iso/iec 20000" OR cmmi-svc OR mps-sv OR "it service management" OR itsm OR "service oriented it management" OR "gestão de serviços de ti") AND (implementation OR implantação OR adoption OR adoção OR "process improvement" OR "melhoria de processo") AND (factor OR fator OR factors OR fatores OR strategy OR estratégia OR strategies OR estratégias). The search string included synonymous terms in Portuguese because we considered Brazilian conferences in this research. The first set of terms refers to the IT service models described in Section “Models for improvement of IT service processes”. Also, it uses the terms “IT service management”, “itsm”, “service oriented it management” and the Portuguese term “gestão de serviços de TI” to collect papers that either do not specifically mention one of these models, or that use other types of models. The second set of terms restricts the results to papers that report cases of the implementation/adoption of such models or cases regarding process improvement using those models. Finally, the third set of terms limits the results to papers that mention factors or strategies in the implementation or improvement of the cited processes. It is worth noting that the terms “(factor OR factors OR strategy OR strategies)” were used by Montoni (2010) for mapping the critical factors in software process improvement, thus being validated in such a context.
The selection of papers was done in three stages:
Selection and preliminary cataloging of the collected sources from the search string;
First relevant selection filter for the publications. This filter was designed by analyzing the abstract and applying the SC1 selection criterion — the paper must provide information about factors that have an influence on the success and failure of the improvement of the IT service processes;
Second relevant selection filter for the publications. This filter was performed by reading the complete text of the paper and applying the following selection criteria: SC2 — the paper must provide evidence that the presented factors originated from the analysis of studies, research, or case reports on improvement initiatives for service processes; SC3 — the paper must provide information about the factors that have an influence on the improvement initiatives for IT service processes, but not on the processes themselves; and SC4 — the paper must not report identical results from another study that have already been selected during the mapping.
After the third selection stage, we obtained a set of relevant papers. However, in order to complement the results obtained, we applied snowballing procedures to add any relevant papers that had not been identified during the exploration of the search string, but which were referenced by or referred to in any of the papers initially identified. Consequently, the following inclusion criteria were applied: IC1 — the paper is referenced or referred to in one of the papers that was retrieved and selected from the three selection stages; and IC2 — the paper satisfies the criteria from the second and third selection stages (SC1, SC2, SC3, and SC4). For each paper considered to be relevant, the following data was extracted: (a) publishing information, which consists of the title, author(s), date published, and publishing source; (b) abstract of the paper; (c) list of factors that positively influence improvement initiatives for IT service processes; (d) a list of factors that negatively influence improvement initiatives for IT services processes; and (e) origin of the identified factors.
After selecting and extracting data from the papers, we analyzed the data using GT procedures. Thus, we managed to perform two types of analysis — qualitative and quantitative. The qualitative analysis resulted in graphs generated through GT procedures and the discussion regarding the research questions, whereas the quantitative analysis resulted in: (a) a list of findings with a positive influence and the total number of occurrences; (b) a list of findings with a negative influence and the total number of occurrences; and (c) the list of critical success factors and the total number of occurrences.
Results from the systematic mapping study and snowballing
After establishing the research protocol, we performed the systematic mapping study in May 2013. During the first selection stage, we used the search string in the Compendex and Scopus search engines. Also, we reviewed the papers from national conferences (SBQS and WAMPS). As a result, we identified 65 papers in Compendex and 54 papers in Scopus (41 papers were available in both digital libraries). None were identified in SBQS or WAMPS.
During the second selection stage, we read the abstract from each of the retrieved papers. Then, using the SC1 selection criterion we selected 14 papers. In the third stage, we read the full text of 13 of the 14 selected papers — one of the selected papers could not be downloaded. From this set of papers, one did not meet the SC3 selection criterion because it only presented factors that influenced the execution of the incident management process. Also, there were another 4 papers which did not meet the SC2 selection criterion, because they derived factors from the COBIT framework controls or a literature review, or did not present a description of the study from which the factors were found. Thus, using a systematic mapping study, we selected 8 papers which met the selection criteria.
Besides this initial set of analyzed papers, we read the title and abstract of papers that were retrieved using snowballing procedures. As a result, 12 papers were selected in the second selection process stage. In the third stage, since one paper was unavailable for download, we read the complete text of 11 of the 12 selected papers. After that, 3 papers did not meet the SC2 selection criterion, because they derived factors from a literature review. Also, one paper was discarded because it did not meet the SC3 selection criterion — it presented factors influencing IT service maintenance and operation — and one paper did not meet the SC4 selection criterion because the content about risk factors was identical to that presented in other papers. Thus, using snowballing, we were able to identify 6 papers which met the IC1 and IC2 inclusion criteria. Table 1 shows the 14 papers (8 using the systematic mapping study and 6 using snowballing) that were selected as a basis for the analysis of critical factors in the improvement of IT service processes.
From this set of papers we identified critical factors in two different approaches. The factors that were derived from empirical studies and researches were derived directly, whereas, if the reviewed paper described a case study of improvement initiatives, besides identifying factors that were explicitly indicated in the paper, we identified further factors from the content of the described cases. Also, some papers did not provide a specific section describing factors, and, therefore, we extracted factors from other sections such as lessons learned, discussions, and conclusions.
Analysis of the results
Using the data obtained from the review we were able to answer the research questions we defined above. We initially analyzed the obtained data using the open coding processes from the GT method. First, we identified quotes from the text of the papers which indicated factors associated with critical success or critical failure. Then, we also identified if there were similarities or differences among these quotes. From this initial analysis, we classified and separated our findings according to the subject and the type of influence it has (positive or negative) on improvement initiatives. These classifications were called “findings with a positive influence” and “findings with a negative influence”, and yielded 33 categories in total. Table 2 and Table 3 show the findings with a positive and negative influence, respectively, and indicate in which paper these findings were located. Additionally, Table 4 and Table 5 present occurrences of some findings.
We performed the axial coding procedure in two iterations in order to establish more abstract categories for the codes of the findings. In the first iteration, the codes for the findings were grouped into 22 categories called “Properties of critical success factors”. These properties group similar findings that differ only by the type of influence (positive or negative). Table 6 shows the occurrences of all properties of critical factors, grouping the 154 occurrences of findings within the papers. This total is the raw data found in the selected papers and initially grouped in the 33 findings. The findings grouped for each property are indicated in the same table. Additionally, during the second iteration, we aggregated the categories of the properties in a more abstract manner. Thus, there was a total of 8 categories of critical success factors that group properties related to a common topic: [Fa01] Processes; [Fa02] Support, commitment and involvement; [Fa03] Internal and external resources; [Fa04] Skills of the people involved in the project; [Fa05] Structure and culture within the organization; [Fa06] Implementation strategy for the improvement project; [Fa07] Collaboration, communication and conciliation of the people involved; and [Fa08] Strategies for the advertisement of the project and the publication of its results. Table 7 shows the number of occurrences of critical success factors and the properties grouped for each factor.
In this analysis it was not necessary to perform the selective coding, since we were able to comprehend the critical success factors after the open and axial coding. In order to perform the selective coding it is necessary to validate all the categories and their relationships until we are able to achieve the theoretical saturation. This saturation indicates that the explanatory power of the theory is approximately null when more evidence is collected (Bandeira de Mello and Cunha,2003). It is also important to note that, in order to perform the selective coding, it is necessary to collect more data, thus it is necessary for more papers to be published in the coming years. Figure 1 summarizes the analysis process.
After identifying all the categories, we counted the number of occurrences in the analyzed papers, aiming to identify which categories were the most cited. In order to establish relationships between the identified categories, we created a graph for each critical success factor. We used the connector “it is a property of” to relate the category codes of properties of critical success factors with the category codes of critical success factors. Also, we used the connectors “it is evidence of the presence of” and “it is evidence of the absence of” to relate the category codes of the findings with the category codes of properties of critical success factors. These connectors show a positive and negative variation, respectively, in the presence of a factor.
Readers must take note that the codes presented in the graph are followed by two numbers which represent the groundness (foundation ground) and density (theoretical density) of the code. The groundness is the number of times that the code was mentioned in the papers, while the density describes the number of relationships that the code has with other codes. Figures 2,3,4, and5 show the graphs with the categories and relationships regarding the factors: [Fa01] Processes; [Fa02] Support, commitment and involvement; [Fa03] Internal and external resources; and [Fa07] Collaboration, communication, and conciliation of the people involved. The figures present the properties related to the factor and the findings with a positive and negative influence that were grouped in each property. For instance, in Figure 1 it can be seen that the type of finding with a positive influence — [Fi16] Existing processes are adequate — and the type of finding with a negative influence — [Fi24] Inadequate processes — show the presence and absence of the property "[P11] Process adequacy", respectively. Furthermore, the properties "[P08] Processes monitoring and control", "[P11] Process adequacy", and "[P18] Adequate order for the implementation of the processes", are related to the factor "[Fa01] Processes".
Comparison of critical factors in software process improvement
When analyzing the critical factors identified by Montoni (2010) for software process improvement, and comparing them with the critical factors for service processes, we managed to identify some similarities and differences. In fact, the factors “Processes”, “Support, commitment and involvement”, and “Skills of the people involved in the project” or ”Skills of the members of the organization”, appear in both studies. In Montoni’s (2010) study, these three factors were considered by the author to be the most important, due to the total number of occurrences in comparison with the other factors. In our study, two criteria were used to determine whether or not a factor is highly important: total number of occurrences and occurrences in all selected articles. Therefore, we considered highly important factors to be either the ones with a high number of occurrences in relation to other factors or the ones which occur in all selected articles. On the other hand, factors of medium or low importance are factors with a medium or low number of occurrences. Therefore, in terms of the number of occurrences, the factors “Processes” and “Support, commitment and involvement” were considered to be highly important, while the factor “Skills of the people involved in the project” would be a factor of medium importance. However, considering its appearance in all reviewed papers, the factor “Support, commitment and involvement” is considered to be the most critical. Also, in the present study, properties regarding skills of the members of the organization who implement the improvement, as well as the skills of the consultants, were grouped in the same factor. However, Montoni (2010) grouped the skills of the external consultants into the factor “Respect for the consultants by the organization’s members” which also includes good relationships and trust between the organization and the consultants. This factor was not structured in our study and the properties related to the relationship among the people involved were grouped in the factor “Collaboration, communication and conciliation of the people involved”. It is worth noting that the factor “Motivation and satisfaction of the members of the organization”, which is indicated in Montoni’s (2010) research, was also not structured in our analysis, since we only verified general properties about the commitment of the employees.
In this paper, the factor “Internal and external resources” was considered to be of low importance due to the total number of occurrences. This factor also considers the use of external consultants in the project. According to Montoni (2010), the factor “Resources” was evaluated as a factor of medium criticality. However, the properties related to the use of consultants were grouped in the factor “Skills of the members of the organization” which was evaluated by Montoni (2010) as a highly important factor.
In both studies there is a factor related to the management of the improvement implementation as a project. The factor “Software process improvement implementation strategy” defined by Montoni (2010) was considered to have a medium level of influence. However, in our study, the factor “Implementation strategy of the improvement project” was considered to be the factor with the greatest influence due to the number of occurrences. Also, in both studies there are factors that relate to the dissemination of the obtained results and the advertisement of the improvement project to the people involved. Thus, the factor “Strategies for the advertisement of the project and the publication of its results” was structured in our research and was considered to be of low criticality, while in Montoni’s (2010) research the factor “Conscientization of the benefits of the implementation of the improvement processes” was considered to have a medium level of influence.
The factor “Structure and culture within the organization” was defined in our study as a factor with a medium level of influence and it is related to the management, structure, planning, and cultural properties of the organization that is implementing the improvement project. Montoni (2010) presented the factors “Organization structure” and “Acceptance towards change”, with a medium and low level of influence, respectively. Such factors are related to similar contexts (with the exception of culture in methodologies and processes, which was considered by the author in the factor “Processes”).
Finally, in this paper, the factor "Collaboration, communication, and conciliation of the people involved" was considered to be of low criticality. The factors "Conciliation of interests" and "Policy for recognition of collaboration in the improvement process", which were defined by Montoni (2010) and considered to have a medium level of influence, contain similar issues (except for the relationship with the external consultants that was grouped by the author in the factor "Respect for the consultants by the members of the organization").
This paper described a study of critical success factors in improvement initiatives for IT service processes. We performed a systematic mapping study and snowballing and applied GT procedures for analysis purposes. By analyzing critical factors, we intend to assist organizations in developing strategies for the implementation of improvements for IT service processes, indicating features that could guide the success of such initiatives.
Some factors were considered to be more relevant and highlight the need for strategies for implementation of the improvement project which involve the execution and governance of the project, the management of benefits and changes, the understanding and the scale of the project, the adequacy of the improvement framework used, and management support and commitment of the people involved in the project.
We also identified other factors with less influence which relate to: organizational structure and culture (business strategy, planning, favorable culture towards the improvement project); experience and commitment of the people involved in the project; resources and team allocation; external consultants and tools of the project; strategies for the advertisement and acceptance of the project, and the dissemination of its results; and collaboration, communication and conciliation between the teams, clients, external consultants, and providers.
In this paper we also compared the identified factors with the critical factors of improvement initiatives for software processes. Despite some differences regarding the degree of importance, we perceived many similarities regarding the related context of the factors. Also, the identified factors indicate similar concerns that could be applied to both types (software and services) of improvement initiatives. However, there are still few reports regarding influencing factors for the improvement of IT service processes. The CMMI-SVC and the MR-MPS-SV are recent (released in 2009 and 2012, respectively) and their adoption, given the required time, could aid in the comprehension and implementation of improvements in these types of processes.
The use of snowballing reinforced the perceived critical success factors from the systematic mapping study and the positive and negative findings from the additional papers were able to be distributed into the 8 existing factor categories. Also, although some factors had a different number of occurrences and even changed between the categories for medium and low importance, the factors of “Processes” and “Implementation strategy of the improvement project” still had the highest number of occurrences after performing the snowballing. Furthermore, the factor “Support, commitment and involvement” continued to be the only one to appear in all selected publications, and its number of occurrences increased, evolving from a medium to a high level of criticality, which confirmed its relevance.
It is worth noting that, as a procedure for guaranteeing the quality of our research, another researcher, with a high level of experience in performing and reviewing similar studies, evaluated the protocol and its execution. We made adjustments and achieved a consensus regarding the performed analyses. Nevertheless, there are threats to the validity of our study. For instance, there are some papers that might not be available in the analyzed sources, and, therefore, might not have been considered during the systematic mapping study or snowballing (e.g., theses and dissertations). This can affect the generalization of our results, generating a threat to the external validity (Wohlin et al.2000) of the study. Also, the composition of the search string can be a threat to the construction validity, since some papers might not contain the chosen terms. Such a threat was minimized by performing the snowballing, because it was not necessary for the selected papers to satisfy the search string. Moreover, using snowballing we could access papers that were indexed in sources other than the Scopus and Compendex digital libraries. Some of the selected papers were indexed in the IEEE, ACM, and DBLP, while others were not indexed in any of the main search bases. Another threat to the conclusion validity (Wohlin et al.2000) is that some papers briefly describe the factors, or just mention them. Thus, having just a brief description of the factors in such papers can influence our understanding and the adequacy of the category to which they might belong. Finally, a threat to the internal validity (Wohlin et al.2000) is the qualitative analysis of the summaries during the first selection stage of the papers, which, if not properly performed, could have resulted in us discarding relevant papers. However, we minimized this threat by performing this analysis from the point of view of two different researchers. In this case, we guaranteed that, in order to discard a paper, the researchers must agree that the provided information does not satisfy the inclusion criteria.
As a future work, we intend to execute a systematic mapping study and snowballing protocol again in the coming years in order to collect more data and achieve the theoretical saturation through GT selective coding. Furthermore, we intend to investigate the perception of the identified factors by the employees of the organizations that use IT service improvement models. Thus, we intend to confirm our findings and verify their applicability. Another future possibility with this research is the definition of a process line for strengthening the software services in software factories. Such a process line would consider the factors identified as critical. We expect that, by using the process line, organizations will be able to derive processes that guide software companies and diminish problems related to the occurrence or the absence of the critical success factors described in this paper.
Bandeira De Mello R, Cunha C: Operationalizing the method of Grounded Theory in Researches of Strategy: techniques and analysis procedures with support of ATLAS/TI software. Curitiba: Paper presented at the 1st Encontro de Estudos em Estratégia; 2003:18–20. May 2003. (in Portuguese) May 2003. (in Portuguese)
Basili VR, Caldiera G, Rombach HD: The Experience Factory. In Encyclopedia of Software Engineering. Edited by: Marciniak JJ. New York: John Wiley & Sons; 1994.
von Bon J: ITIL - A Pocket Guide - 2011 Edition. Netherlands: Van Haren Publishing; 2011.
Cater-Steel A: IT Service Departments Struggle to Adopt a Service-Oriented Philosophy. International Journal of Information Systems in the Service Sector 2009, 1: 69–77.
Cater-Steel A, Pollard C: Conflicting views on ITIL implementation: managed as a project – or business as usual? Paper presented at Information Resources Management Association International Conference. Ontario: IRMA; 2008. May 2008 May 2008
Cater-Steell A, Tan W-G: Implementation of IT Infrastructure Library (ITIL) in Australia: Progress and success factors. Auckland: Paper presented at IT Governance International Conference; 2005. Nov 2005 Nov 2005
Cater-Steel A, Toleman M, Tan WG Paper presented at the 17th Australasian Conference on Information Systems. In Transforming IT service management- The ITIL impact. Adelaide: ACIS; 2006:6–8. Dec 2006 Dec 2006
Cartlidge A, Hanna A, Rudd C, Macfarlane I, Windebank J, Rance S: An introductory overview of ITIL v3. United Kingdom: IT Service Management Forum; 2007. http://itsmfi.org/files/itSMF_ITILV3_Intro_Overview.pdf
Coleman G, O’Connor R: Investigating software process in practice: A grounded theory perspective. J Syst Software 2008, 81: 772–784. 10.1016/j.jss.2007.07.027
Diirr T, Santos G: Critical Success Factors for Information Technology Services Processes Improvement: A Systematic Mapping. Salvador: Paper presented at the 12th Simpósio Brasileiro de Qualidade de Software – SBQS; 2013:1–5. Jul 2013. (in Portuguese) Jul 2013. (in Portuguese)
Ghayekhloo S, Sedighi M, Nassiri R, Latif SG, Tirkolaei HK: Pathology of organizations currently implementing ITIL in developing countries. In: Proceedings of the 2nd International Conference on Computer and Electrical Engineering. Dubai: ICCEE; 2009:28–30. Dec 2009 Dec 2009
Hochstein A, Brenner W: Implementation of service-oriented IT management: An empirical study on Swiss IT organizations. Paper presented at International Conference on Service Systems and Service Management. Troyes: ICSSSM; 2006:91–97. Oct 2006 Oct 2006
Hochstein A, Tamm G, Brenner W Proceedings of the 13th European Conference on Information System. In Service-oriented IT management: benefit, cost and success factors. Regensburg: ECIS; 2005:26–28. May, 2005 May, 2005
ICASA: COBIT 5 – Control Objectives Management Guidelines Maturity Models: A Business Framework for the Governance and Management of Enterprise IT. Information Systems Audit and Control. USA: Association; 2012.
Iden J: Implementing IT Service Management: Lessons Learned from a University IT Department. In Information Technology Governance and Service Management: Frameworks and Adaptations. 1st edition. Edited by: Cater-Steel A. New York: Information Science Reference, Hershey; 2009.
Iden J, Langeland L: Setting the Stage for a Successful ITIL Adoption: A Delphi Study of IT Experts in the Norwegian Armed Forces. Information Systems Management 2010, 27: 103–112. 10.1080/10580531003708378
ISO/IEC: ISO/IEC TR 20.000–3: Information Technology – Service Management – Part 3: Guidance on scope definition and applicability of ISO/IEC 20.000–1. Switzerland: International Standard Organization/International Electrotechnical Commission; 2009.
ISO/IEC: ISO/IEC TR 20.000–4: Information Technology – Service Management – Part 4: Process reference model. Switzerland: International Standard Organization/International Electrotechnical Commission; 2010a.
ISO/IEC: ISO/IEC TR 20.000–5: Information Technology – Service Management – Part 5: Exemplar implementation plan for ISO/IEC 20.000–1. Switzerland: International Standard Organization/International Electrotechnical Commission; 2010b.
ISO/IEC: ISO/IEC 20.000–1: Information Technology – Service Management – Part 1: Service management system requirements. Switzerland: International Standard Organization/International Electrotechnical Commission; 2011.
ISO/IEC: ISO/IEC 20.000–2: Information Technology – Service Management – Part 2: Code of practice. Switzerland: International Standard Organization/International Electrotechnical Commission; 2012.
Jalali S, Wohlin C Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement. In Systematic Literature Studies: Database Searches vs. Backward Snowballing. Sweden: ESEM; 2012:19–20. Sept 2012 Sept 2012
Jäntti M: Lessons Learnt from the Improvement of Customer Support Processes: A Case Study on Incident Management. In Product-Focused Software Process Improvement. 10th International Conference, PROFES 2009, Oulu, Finland, Jun 15–17, 2009. Lecture Notes in Business Information Processing, vol 32. Edited by: Bomarius F, Oivo M, Jaring P, Abrahamsson P. Heidelberg: Springer; 2010:317.
Junior V, Andrade J: Critical Success Factors and Benefits of ITIL Model Adoption in a Telecommunication Company. Paper presented at 7th Simpósio de Excelência em Gestão e Tecnologia. Resende, Out: SEGeT; 2010. in Portuguese in Portuguese
Kitchenham BA: Procedures for Performing Systematic Reviews. In Keele University Technical Report TR/SE-0401 and NICTA Technical Report 0400011T.1. Australia: Software Engineering Group Department of Computer Science and Empirical Software Engineering National ICT; 2004. Ltd. . Accessed 17 Sept 2013 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.122.3308 Ltd. . Accessed 17 Sept 2013
Kitchenham BA, Budgen D, Brereton OP: The value of mapping studies - A participant-observer case study. Keele University, UK: Paper presented at 14th International Conference on Evaluation and Assessment in Software Engineering (EASE); 2010:12–13. April 2010 April 2010
Magalhães IL, Pinheiro WB: IT service management in practice: an approach based on ITIL. São Paulo: Novatec; 2007. (in portuguese) (in portuguese)
Montoni MA: A research on critical success factors in software processes improvement initiatives. Universidade Federal do Rio de Janeiro - UFRJ: PhD Thesis; 2010. in Portuguese in Portuguese
Montoni MA, Rocha ARC: Applying Grounded Theory to Understand Software Process Improvement Implementation. Portugal: Paper presented at 7th International Conference on the Quality of Information and Communications Technology - QUATIC 2010; 2010. 17 Sept - 2 Oct 2010 17 Sept - 2 Oct 2010
Montoni MA, Rocha ARC: Using Grounded Theory to Acquire Knowledge About Critical Success Factors for Conducting Software Process Improvement Implementation Initiatives. International Journal of Knowledge Management 2011, 7: 43–60.
Pollard C, Cater-Steel A: Justifications, strategies, and critical success factors in successful ITIL implementations in U.S. and Australian companies: An exploratory study. Inf Syst Manag 2009, 26(2):164–175. 10.1080/10580530902797540
Rockart JF: Chief executives define their own data needs. Harv Bus Rev 1979, 57(2):81–93.
Santos G, Kalinowski M, Rocha AR, Travassos GH, Weber KC, Antonioni JA: MPS.BR Program and MPS Model: Main Results, Benefits and Beneficiaries of Software Process Improvement in Brazil. Lisbon: Paper presented at 8th International Conference on the Quality of Information and Communications Technology – QUATIC 2012; 2012:3–6. Sept 2012 Sept 2012
SEI: CMMI for Services - CMMI-SVC. Pittsburgh: Software Engineering Institute; 2010.
Sharifi M, Ayat M, Rahman AA, Sahibudin S: Lessons learned in ITIL implementation failure. Kuala Lumpur: Paper presented at the International Symposium on Information Technology – ITSim; 2008:26–28. Aug. 2008 Aug. 2008
SOFTEX: Reference model for services process improvement. Brasil: Associação para Promoção da Excelência do Software Brasileiro; 2012. . (in Portuguese) http://www.softex.br/mpsbr . (in Portuguese)
Strauss A, Corbin J: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. London: SAGE Publications; 1998.
Tan W-G, Cater-Steel A, Toleman M: Implementing it service management: A case study focussing on critical success factors. J Compu Info Syst 2009, 50(2):1–12.
Tan WG, Cater Steel A, Toleman M, Seaniger R: Implementing centralised IT service management: Drawing lessons from the public sector. Paper presented at 18th Australasian Conference on Information Systems. Toowoomba: ACIS; 2007:5–7. Dec 2007 Dec 2007
Wan J, Liang L: Risk Management of IT Service Management Project Implementation with Killer Assumptions. Technol Invest 2012, 3: 48–55. 10.4236/ti.2012.31007
Wan J, Zhu S, Wang Y: Empirical analysis on risk factors of IT service management project implementation. Dalian: Paper presented at the 4th International Conference on Wireless Communications, Networking and Mobile Computing – WiCOM; 2008:12–14. Oct 2008 Oct 2008
Wohlin C, Runeson P, Höst M, Ohlson M, Regnell B, Wesslén A: Experimentation in Software Engineering: An Introduction. Norwell - USA: Kluwer Academic Publishers; 2000.
The authors declare that they have no competing interests.
A uthors’ contributions
The research described in this paper is part of TD’s MSc dissertation. GS is TD’s advisor. All authors participated in the definition of the systematic mapping study and the snowballing described in this paper.TD conducted the study and GS provided guidance and also reviewed the result in order to guarantee its quality. All authors participated in the writing of this paper. All authors read and approved the final manuscript.