Towards guidelines for building a business case and gathering evidence of software reference architectures in industry

1.1 Research problem

Although the adoption of an RA might have plenty of benefits for an organization, it also implies several challenges, such as the need for an initial investment (Martínez-Fernández et al. [2013b]) and getting real evidence for driving its design and use (Angelov et al. [2013]). Currently, organizations have little support for dealing with these two challenges.

First, there is a shortage of economic models to precisely evaluate the benefit of RAs in order to make informed decisions about their adoption in an organization (Martínez-Fernández et al. [2013b]). Organizations with a wide portfolio of applications, which may consider adopting an existing or new RA to create and maintain such applications, lack an approach to know whether it is worth for them to invest on the adoption of an RA. This situation could be addressed by making a business case with the help of an economic model that perform cost-benefit analysis about the adoption of an RA (Reifer [2002]).

Second, there is also a shortage of experience reports disseminating the context of RAs in industry. For instance, a recent literature review about evidence in software architecture, in which only two papers were about RAs, shows that there is limited knowledge about RAs (Qureshi et al. [2013]). As a result, practitioners usually find the current literature about RA scarce and abstract (Angelov et al. [2012]), limiting the industrial uptake of research results in the field. In this context, we argue that in order to enable practitioners to fully exploit the benefits of RA adoption and usage, the research community must clarify the context of RA in practice. We propose conducting empirical studies to accumulate real evidence for understanding the context of RAs from essential types of stakeholders. Such evidence aims to help practitioners to better understand RA-related practices and, then, to identify the current challenges to improve these practices in their organization. Throughout the paper, we use the term “RA-related practices” to refer to common practices in RA projects, such as defining the goals of an RA, RA design, RA review, RA use and so on.

1.2 Research goals and contribution

In this context, the goal of this research is to support organizations (i.e., software companies, information technology consulting firms) to deal with the following Research Questions (RQ):

1.3 Paper structure

The paper is structured as follows. In Section 2, we explain the research methodology used. In Section 3, we report the details of how RQ 1 was approached and its results. In Section 4, we report the details of how RQ 2 was approached and its results. In Section 5, we report how the results from RQ 1 and RQ 2 have been packaged to answer the RQ 3, and we present the guidelines for building a business case and gathering evidence of RAs and their context of use. Finally, in Section 6, we end up with conclusions and future work.

The work presented in this paper is a follow-up of the work presented in (Martínez-Fernández et al. [2013a]). It substantially extends such work by:

2.1 Action research

This research consists of an ongoing action research initiative among the GESSI research group and everis. Action research is “learning by doing” - a group of people identify a problem, do something to resolve it, see how successful their efforts are, and if not satisfied, try again (Brydon-Miller et al. [2003]). The action research cycle consists of: 1) diagnosis of a problem, 2) examination of options to solve the problem, 3) selection of options and execution, 4) analysis of the results, and, 5) repetition for improvement.

However, the idea behind this research is not only to help everis to cope with RQ 1 and RQ 2. With the goal of RQ 3 in mind, we also aim to package the solution in such a way that it can be easily used by other organizations dealing with similar problems as everis. The guidelines that will be proposed in this paper are shaped throughout our involvement with everis to cope with the RQ 1 and the RQ 2.

2.2 Context of RAs in everis

We analyze the context of RA projects from our experience with everis. We focus on the case in which everis has designed an RA with the purpose of deriving concrete SAs for each application of a client organization. This usually happens when everis is regularly contracted to create or maintain information systems in client organizations. Each information system of the client organization is built upon the RA and includes many software applications (see Figure 2). RAs enable reuse of architectural knowledge and software components (normally associated to particular technologies) for the design of concrete SAs in client organizations. Therefore, RAs provide a baseline that facilitates standardization and interoperability as well as the attainment of business goals during applications’ development and maintenance.

Besides, a special characteristic of everis is its previous experience in multiple RA projects. This experience allows everis to build a more abstract industry RM. This RM includes best practices from previous successful experiences, which serve as a reference for new RAs that inherit a certain level of quality.

The context of everis is very similar to other IT consulting firms. As a recent Gartner report shows, IT consulting firms “leverages industry-specific or industry reference models to accelerate client delivery and ensure quality and consistency across client engagements” (Brand [2014]). However, “clients must ensure that generic industry or reference models [...] are sufficiently customized and tailored to enable their unique business capabilities and environments” (Brand [2014]), so that the RM does not stifle competitive advantage of the RA.

Next subsections respectively show the types of projects that can be found on the everis context and the stakeholders that participate in each project.

Types of projects

Stakeholders for RA analysis

Stakeholders need to be clearly defined for RA assessment purposes (Angelov et al. [2008]). The people involved in an RA assessment are the evaluation team, which conducts the empirical studies of the guidelines, and stakeholders from architectural projects. In the three types of projects defined above performed by IT consulting firms, we consider the following five stakeholders essential for RA assessment: project business manager, project technological manager, software architect, architecture developer, and application builder. Each of these stakeholders has a vested interest in different architectural aspects, which are important to analyze and reason about the appropriateness and the quality of the three types of projects (Gallagher [2000]). However, there could be more people involved in an architectural evaluation, as Clements et al. indicate in ([Clements et al. 2001]). As a consequence, although this context is generic for IT consulting firms, projects’ stakeholders may vary between firms. Below, we describe to which type of project essential stakeholders belong and their interests.

RM project It is composed of software architects from the IT consulting firm that worked in previous successful RA projects. They are specialized in architectural knowledge management. Their goal is to gather the best practices from previous RA projects’ experiences in order to design and/or improve the corporate RM.

RA projects RA projects involve people from the IT consulting firm and likely from the client organization. Their members (project technological managers, software architects and architecture developers) are specialized in architectural design and have a medium knowledge of the organization business domain.

Project technological managers from the IT consulting firm are responsible for meeting schedule and interface with the project business managers from the client organization.

Software architects (also called as RA managers) usually come from the IT consulting firm, although it may happen that the client organization has software architects in which organization’s managers rely on. In the latter case, software architects from both sides cooperatively work to figure out a solution to accomplish the desired quality attributes and architecturally-significant requirements.

Architecture developers come from the IT consulting firm and are responsible for coding, maintaining, integrating, testing and documenting RA software components.

SA projects Enterprise application projects can involve people from the client organization and/or subcontracted IT consulting firms (which may even be different than the RM owner) whose members are usually very familiar with the specific organization domain. The participation of the client organization in RA and SA projects is one possible strategy for ensuring the continuity of their information systems without having much dependency on subcontracted IT consulting firms.

Project business managers (i.e., customer) come from client organizations. They have the power to speak authoritatively for the project, and to manage resources. Their aim is to provide their organization with useful applications that meet the market expectations on time.

Application builders take the RA reusable components and instantiate them to build an application.

3.1 First cycle: survey to check existing value-driven data in RA projects

In this survey, a sample of 5 everis’ RA projects and 5 SA projects were selected on the basis of their suitability and feasibility to contact at least with one person that participated in the projects.

3.2 Second cycle: a case study to apply an economic model to calculate the ROI of adopting an RA

In this case study, a sample of 1 everis’ client organization RA project in a public administration and 1 SA project was selected. These projects were selected because the public administration that adopted the RA was interested in the study results. Besides, by the time we conducted the study, everis’ started the aforementioned SA project, being highly feasible to collect quantitative data. Although we were aware of other SA projects with participants that do not belong to everis, it was not possible to contact with them.

Results from the previous survey (see Section 3.1) revealed that the data available in order to calculate costs and benefits were effort and software metrics. We collected these metrics, which are presented in (Martínez-Fernández et al. [2013b]), from two types of tools. In order to collect data, JIRA (Atlassian [2014]) was used to collect the invested effort from training, development and maintenance activities. Keeping track of activities is common in practice for project management and auditing. In addition to that, Sonar (SonarSource [2014]) was used to gather software metrics to analyze the benefits that can be found in the source code. Sonar offers tool support for obtaining general software metrics such as LOC, dependencies between modules, technical debt, and percentages of tests and rules compliance.

3.2.1 Results

After retrospectively collecting the data of the past RA and SA projects, we analyzed which cost-benefit factors of adopting an RA could be calculated and envisaged the REARM economic model (Martínez-Fernández et al. [2013b]) in order to calculate the ROI of the RA investment. By the time we conducted the study, the public organization had already: (1) adopted an RA designed by everis, (2) created an application using the RA, and (3) fixed errors discovered in the RA software elements that were reused by the application.

We calculated a three-year ROI of 42% with a payback period of 16,5 months and 7 applications in the public organization (see Figure 4).

Moreover, applications are introduced into the market earlier from the seventh month on. This is due to the effort avoidance of reusing the RA in the development of new applications.

The results of this study are further reported in (Martínez-Fernández et al. [2013b]).

4.1 First cycle: identifying practical review criteria for RAs

In order to gather evidence of RAs, it becomes necessary to previously identify the aspects that are relevant for practitioners about RAs. In this section we identify an initial set of criteria, which might be further refined after gathering evidence from RAs. To do so, we analyzed the key criteria mentioned by everis’ software architects during the meetings and focus group discussions of our action research initiative, and we also studied the literature. We prioritized the everis’ vision to make a more practitioner-oriented set of criteria.

Although a commonly accepted set of criteria to evaluate RAs does not exist (Angelov et al. [2008]; Gallagher [2000]; Graaf et al. [2005]), it has been claimed that RAs have to be evaluated for the same aspects as SAs (Angelov et al. [2008]). For this reason, we started by analyzing some available works on SA evaluation (Bass et al. [2008]; Clements et al. [2001]; Falessi et al. [2010]). However, existing evaluation methods for SAs are not directly applicable to RAs or other architecture-centric approaches such as product lines architectures because they do not cover their generic nature (Angelov et al. [2008]). The development of a family of software systems has some characteristics that distinguish it from the development of software systems (Montagud et al. [2011]). Therefore, existing evidences for product line architectures can be also used to evaluate RAs, namely: generic characteristics such as “variability, reusability, commonality, and compositionality” (Montagud et al. [2011]); the propagation of architectural decisions while reusing common assets (Montagud et al. [2011]); and lower development costs with respect to developing systems individually (Ali et al. [2009]; Linden et al. [2004]; Montagud et al. [2011]).

Aspect 1 refers to the need of determining the context and classifying an RA. As Angelov et al. point out (Angelov et al. [2012]), there are five types of RAs depending on their characteristics. Among the most important characteristics are (Angelov et al. [2012]):

The classification of an RA is vital to better understand its limits, to ensure its congruency, and to facilitate its evaluation.

Aspect 2 consists of the quality attributes targeted by an RA. The achievement of quality attributes is in fact an SA most compelling reason for existence (Bass et al. [2003]). However, SAs and RAs do not strictly determine all of an application’s qualities. One example is usability: “whether the user sees red or blue backgrounds, a radio button or a dialog box” (Clements et al. [2001]) is not determined by an SA or an RA. A list of quality attributes that lie squarely in the realm of SAs is defined by the architecture tradeoff analysis method (Clements et al. [2001]): performance, reliability, availability, security, modifiability, portability, functionality, variability, subsetability and conceptual integrity. For instance, variability shows how well an RA could be expanded or modified to produce new SAs of applications. Besides, an RA could address more architectural qualities than an SA (e.g., variability, reusability, commonality, compositionality, and applicability) (Angelov et al. [2008]; Montagud et al. [2011]). Quality attributes analysis should be wider for RAs in this sense.

Aspect 3 comprises architectural decisions. Many prominent researchers (Angelov et al. [2008]; Falessi et al. [2010]) highlight the importance of architectural decisions for the SA design process and the architectural evaluation. For RAs, architectural decisions are even more important than in a single software system since, owing to systematic reuse, an inadequate design decision could be propagated to several software systems (Montagud et al. [2011]).

Aspect 4 consists of the supportive technologies such as methods, techniques and tools (Falessi et al. [2010]; Nakagawa et al. [2011]) that aim to improve the RA design process and support the use of the RA during the development of applications. Moreover, this aspect is very important for practitioners, since they are interested in knowing the latest versions of technologies and tools used in the RA projects, and providing application builders with tools that improve their productivity.

Aspect 5 refers to business qualities of an RA. SAs also address business qualities (Angelov et al. [2008]) (e.g., cost, time-to-market) that are business goals, i.e. the objectives of an organization that affect their competence (Bass et al. [2008]). These business qualities are even more important in the context of families of applications, such as RAs: “the main arguments for introducing software product family engineering are to increase productivity, improve predictability, decrease the time to market, and increase quality (dependability)” (Linden et al. [2004]).

We recommend gathering evidence about these five aspects, which are summarized in Table 2, in order to improve RA-related practices. Next, Section 4.2 explains how we are gathering evidence about these aspects. Currently, there are no guidelines to support the gathering of these aspects altogether. This has motivated our work.

4.2 Second cycle: surveys to gather evidence to improve RA-related practices

In these surveys, the target populations were RA projects and SA projects executed by everis. A sample of 9 representative everis’ projects in client organizations were selected. All these projects were from Europe (7 from Spain).

In order to collect data, semi-structured interviews were used for project technological managers, software architects, and client’s project business managers. The reason of using interviews is that these roles have higher knowledge than the other roles about the architectural aspects of the Table 2, or another perspective in the case of client’s project business managers, so we wanted to collect as much information as possible from them. Prior to the interviews, questionnaires were delivered to collect personal information about the interviewee and to inform him/her about the interview. On the other hand, online questionnaires were used for architecture developers and application builders, since most of their questions are about supportive technologies and their responses can be previously listed, simplifying the data collection process.

To perform data analysis, the research team held several discussion meetings during and after data collection, and established specific protocols and templates for data analysis. In the case of the interviews, we used an Excel-based template to organize each participant’s answer to each question using tables. For doing this, we used the interview transcripts and individual notes taken by different researchers during the interviews. Then, we analyzed the data from two perspectives that lead us to a better understanding and interpretation of the results. First, we analyzed the answers at the project level, in order to understand the specific context of each project and the perspective of its stakeholders. Second, we analyzed the answers at whole by assessing all participants’ answers related to each question, in order to categorize their answers using template tables that described (in each column): the name of the category, a detailed description of the cases covered by the category, the participant, and the explicit sentences that support the category. Such categories were then further discussed and analyzed by the entire team in order to better analyze the evidence and improve our understanding until reaching an agreement. As a result, some categories were split, modified, discarded or added to ensure that all answers were well-represented.

5.1 Formative stage: shaping the guidelines

In the formative stage, the goal is to incrementally shape the guidelines from the feedback of the action research. In order to analyze whether other organizations deal with similar problems as everis, we highlight the similarities of RAs designed by everis with other RA contexts that have been reported in the literature and by practitioners. After this analysis, we will only package into the guidelines the material that could be used under other organizations context.

5.1.2 Packaging the guidelines

everis’ results are being suitably packaged with the aim of being applied in prospective RA projects and also in similar organizations. Figure 6 summarizes the guidelines for building a business case and gathering evidence of RAs in industry.

First of all, organizations that may want to use these guidelines need to fit into the context depicted in Section 5.1.1. This means that they need to design an RA based on practical experience, and to use such RA for the development and maintenance of a family of applications in industry. This is because the input for using the guidelines is evidence from real RA projects.

The guidelines support organizations:

1. 1.

   To decide whether to invest on an RA by providing them with:

   • A checklist to analyze existing value-driven data in RA projects.

   • An economic model that uses such value-driven data to calculate the ROI of adopting an RA.

    
2. 2.

   To improve RA-related practices based on corporate evidence by providing them with:

   • A practitioner-oriented set of criteria to understand and evaluate an RA.

   • Templates of interview guides and online questionnaires to gather relevant evidence.

    

On the one hand, to analyze whether it is worth to invest on an RA, a checklist of value-driven data that an organization might have is facilitated to check if the provided economic model provided can be executed. On the other hand, to improve RA-related practices with evidence, a set of relevant aspects for RAs is facilitated to check which ones are important for an organization and then use the provided template surveys to gather evidence.

To gather evidence with the help of the aforementioned artifacts, empirical studies can be set up and carried out. Several data collection techniques exist (Lethbridge et al. [2005]). We propose surveys that use interviews and questionnaires, case studies that require documentation analysis and metrics gathering, and focus groups. Below, we explain the dimension, context, objective and method applied in each empirical study of the guidelines. To see the threats to validity in everis, the reader is referred to (Martínez-Fernández [2013]).

I) A survey to check existing value-driven data in RA projects

Dimensions

RQ 1.

Context

Typically, organizations do not have resources to compare the real cost of creating applications with and without an RA and historical data may be scarce. Thus, alternatives should be considered.

Objective

The objective of this survey is to identify the quantitative information that can commonly be retrieved in RA projects in order to quantitatively calculate the costs and benefits of adopting an RA in an organization. This is an initial step to create repeatable techniques for performing a cost-benefit analysis.

Method

Exploratory surveys with personalized questionnaires applied to relevant stakeholders (e.g., manager, architect, developer) to find out the quantitative data that has been collected in RA projects and SA projects. An example of conducting this empirical study and its approach for data collection is described in Section 3.1.

II) A case study to apply REARM to calculate the ROI of adopting an RA

Dimensions

RQ 1.

Context

Before deciding to launch an RA, organizations need to analyze whether to undertake or not the investment. Offering organizations an economic model that is based on former RA projects data can help them to make more informed decisions.

Objective

The objective is to analyze whether it is worth investing on an RA with the help of an economic model, in order to improve the communication among architects and management, and to improve their decisions.

Method

A case study that applies an economic model to calculate the ROI of adopting an RA. Depending on the maturity of the organization, two approaches can be applied. If the organization does not have experience with an RA, the economic model should be fed with estimated data. Nevertheless, when the organization already has experience with RAs (i.e., the case of IT consulting firms), real data can be gathered by means of an exploratory quantitative post-mortem analysis. Then, the economic model quantifies the potential advantages and limitations of using an RA. Some related works explain how to calculate the ROI of a product (Forrester [2014]), software reuse (Frakes and Terry [1996]; Poulin [1997]), and software product lines (Ali et al. [2009]). We suggest the use of REARM, which is an economic model specific for RAs presented in (Martínez-Fernández et al. ([Martínez-Fernández et al. 2013b])). An example of conducting this empirical study and its approach for data collection is described in Section 3.2.

III) A focus group to study the relevant criteria of RAs for an organization

Dimensions

RQ 2.

Context

Typically, organizations drive the design and use of RAs in an unsystematic manner (Eklund et al. [2005]). To drive RA-related practices based on evidence, it becomes fundamental to identify the relevant aspects of RAs as seen by practitioners.

Objective

The objective of this study is to identify the aspects that are important for each organization in order to support RA-related practices.

Method

A focus group with relevant stakeholders (e.g., manager, architect, developer) to find out which aspects of RAs are important to them. A focus group is considered a proven and tested technique to obtain the perception of a group of selected people on a defined area of interest (Lethbridge et al. [2005]). An example of conducting this empirical study and its approach for data collection is described in Section 4.1.

IV) Surveys to gather evidence to improve RA-related practices

Dimensions

RQ 2.

Context

To reuse architectural knowledge and improve RA-related practices in prospective RA projects, organizations need to understand RA’s characteristics, as well as its potential benefits and limitations. Gathering evidence from previous RA projects is a feasible way to start gaining such an understanding.

Objective

The purpose of the survey is to understand the impact of using RAs for designing the SAs of the applications of an information system of a client organization. This is a descriptive survey that measures what occurred while using RAs rather than why. The following questions are important for organizations in order to understand relevant Aspects 1 to 5 of RAs (defined in Table 2):

1. 1.

   Why is an RA adapted for creating SAs of the organizations’ applications? What type of RA is being designed and used in the organization?

    
2. 2.

   What is the state of practice in requirement engineering for RA projects in the organization?

    
3. 3.

   What is the state of practice in architectural design for RA projects in the organization?

    
4. 4.

   Which tools and technologies are currently being used in RAs projects by the organization?

    
5. 5.

   How does the adoption of RAs provide observable benefits to the different actors involved in RA projects in the organization?

    

Method

Exploratory surveys with personalized questionnaires applied to relevant stakeholders (e.g., architects, developers) to gather their perceptions and needs. An example of conducting this empirical study and its approach for data collection is described in Section 4.2.

Finally, the output for using guidelines for RQ 1 is a business case that evaluates whether it is worth or not to invest on an RA. On the other hand, the output of gathering evidence about Aspects 1 to 5 of Table 2 is a corporate knowledge base about these aspects. It is important to note that, although it is not strictly necessary, it is recommended to gather Aspects 1 to 5 for building the business case. In other words, guidelines are complementary and support each other (e.g., results from a preceding study can be used to corroborate or further develop other results). For instance, in our industry-academia collaboration, the qualitative results about the benefits and drawbacks of RAs supported the unquantifiable benefits, and uncertainties and risk of the business case.

5.2 Summative stage: validating the guidelines

The summative stage will take place once the guidelines will have been adequately shaped and improved. The primary role of this stage will be to validate the version of the guidelines after applying it in everis with more practitioners. This evaluation consists of the use of the guidelines in other organizations to design and conduct empirical studies. Organizations analyzing whether to make the strategic move to RA adoption and organizations that face the design and use of RAs based on evidence will benefit from these guidelines.