Measuring the performance of project risk management: a preliminary model
Article Category: Research Paper
Published Online: Oct 02, 2019
Page range: 1984 - 1991
Received: Nov 21, 2018
Accepted: Mar 18, 2019
DOI: https://doi.org/10.2478/otmcj-2019-0005
Keywords
© 2019 Alfredo Federico Serpell et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Construction industry faces major challenges in relation to the performance of its projects (Hillson, 2009; Zhang, 2016). Uncertainty accompanies globalization and technological advances, which can become risks in projects. If this situation occurs, these risks could produce impacts with negative consequences for the fulfillment of the objectives of projects, such as cost, schedule, scope, and quality (Howard and Serpell, 2012; Serpell et al., 2014; Wolbers, 2009; Zou et al., 2007). For this reason, it is necessary to prevent the occurrence of risks or being prepared with good risk-mitigating responses if they do occur. In addition, there is a need for learning from risk management so that better responses can be applied to new projects.
However, no study has been found in the literature that addresses the problem of how to monitor and control the performance of risk management effectively. Project risk management (PRM) is considered an activity of the planning stage of the project and there is a lack of post facto analysis of how it really worked. Then, this paper raises the importance of addressing the issue of measuring the performance of risk management in construction projects. It aims to establish a different perspective of risk management focused not only on the analysis stage but also on the end of a project. This allows to evaluate what risks occurred, which ones were previously identified and which ones not, what really happened with the risks that occurred, what were the impacts they produce, and how effective was the application of risk responses (Baloi and Price, 2003; Marcelino-Sádaba et al., 2014).
The paper begins with a review of the literature, addressing the issues of how to evaluate the performance of risk management and why it is important to do this for construction projects. Subsequently, a brief description of the research and the model for measuring the performance of PRM are discussed preliminarily. The paper ends with the conclusions and a brief discussion of the following research stages to achieve the final objective of this study.
As defined by Melnyk et al. (2014), performance measurement is an instrument that helps to quantify the efficiency or effectiveness of an action, which provides information on what is happening regarding that action. Different organizational areas use performance measurement (Franco-Santos et al., 2012). The main concern for many organizations is to achieve the strategic objectives defined by them, in addition to measuring the effectiveness of the established processes (Hubbard, 2010). In order to achieve this, there are several approaches, such as the performance measurement system (PMS). This system defines key performance indicators (KPIs), which allow evaluating and measuring the achievement of objectives, identifying the type of measurements, establishing the mechanisms to obtain the data, describing and analyzing the results, and establishing the necessary actions that shall be made in order to improve (Clive, 2014; Taylor and Taylor, 2014; Kennerly and Neely, 2004).
Among the most used tools for measuring the performance is the Balanced Score Card, developed by Kaplan and Norton in 1993. Because of its ease of application and the information it provides, it is widely used among organizations. Zaleha et al. (2012) adapted this tool to measure the performance of enterprise risk management in organizations.
Since organizations set strategic objectives, measuring their performance becomes a fundamental part of defining whether these objectives are being achieved. This is a permanent activity within organizations for closing gaps, identifying opportunities, establishing mechanisms to solve problems, and improving processes continuously. This is particularly important for those processes that are critical for the success of projects (Corona-Suárez et al., 2014).
This paper is concerned with measuring the performance of one of these processes, PRM, considered so important for construction projects. Given its relevance, it is important to understand the behavior of risks to try to handle them in an appropriate way in order to minimize their impact (Zhang, 2016; Szymański, 2017; Khameneh et al., 2016).
Given the current uncertainty and the last global economic crisis of 2008, risk management has gained more importance and has revealed the deficiency of its application at both organizational and project levels (Hubbard, 2009; Mikes and Kaplan, 2013; Pimchangthong and Boonjing, 2017). Organizations such as the Project Management Institute (PMI), the International Organization for Standardization (ISO), and the Australia’s Standard Organization as well as academics, professionals, experts, governments, etc. have expressed their concerns about this issue. As a result, standards for its application, procedures, tools, techniques, and some computing programs that support the application of risk management for a better functioning of enterprises and projects have been developed (Zhang, 2016). In general, the main operational concerns of risk management are on the following topics: 1) the identification of risks; 2) risk analysis and evaluation; 3) the development of techniques, tools, and software that help with the application of risk management; and 4) the evaluation of the risk management maturity within organizations (Lee et al., 2009; Tohidi, 2011).
Several authors have investigated about risk and risk management, raising the interest from different industries and organizations (Hubbard, 2009; Aven, 2010; Hillson, 2007; Ward and Chapman, 2003; Aven, 2016). The risk management process includes five main stages (Baloi and Price, 2003; Hillson, 2007; Ward and Chapman, 2003; Banaitienė et al., 2011; Carter and Chinyio, 2010; Dikmen et al., 2008; ISO, 2009 ; NASA, 2011; Osipova and Eriksson, 2013; Papadaki et al., 2010; Ren et al., 2014; Standards Australia, 2007). These stages are as follows: 1) risk management planning; 2) risk identification; 3) risk analysis and evaluation; 4) response plan to risks; and 5) monitoring and control of risks. In addition, these stages can be divided into two main pillars: 1) identification and analysis, including the stages of risk management plan, risk identification, and risk analysis and evaluation and 2) responses and control, including the stages of response and monitoring plan and control (Corona-Suárez et al., 2014; Washington State Department of Transportation, 2014).
Hubbard (2009), Khameneh et al. (2016), and Zhang (2016) noted that the greater emphasis of the application of risk management has focused mainly on the risk identification and analysis stages. There is a lack of connection between these stages and the risk response plan, monitoring, and control part. This generates a lack of knowledge about the effectiveness of the strategies to face risks and about the impacts of these strategies on project results, because, as indicated by Szymański (2017), an effective risk management does not mean the removal of risk.
Regarding the analysis stage, the quantitative analysis of risk has started with Program Evaluation and Review Technique (PERT). This technique was developed for the Program Evaluation Branch of the Special Projects Office of the US Navy in order to measure and control development progress for the Polaris Fleet Ballistic Missile Program (Malcom et al., 1959). The key goal of the PERT analysis is to generate a distribution of the project duration (Hajdu and Bokor, 2016). Since 1963, the Monte Carlo simulation was used to solve PERT networks more precisely (Van Slyke, 1963), but just recently replaced the analytical solutions for modeling the effects of temporal risks on project duration. Although PERT is a well-known methodology, it has received several critics regarding the use of the beta distribution for activity duration. A study using hypothetical and real-life example projects shows that the accuracy of the three-point estimation is more important than the type of the activity duration distribution (Hajdu and Bokor, 2016).
In the construction industry, risk management is considered a bureaucratic and tedious activity usually applied only at the beginning of the life cycle of the project and merely regarding risk identification and analysis. The nature of projects that this industry carries out is usually characterized by a great deal of uncertainty and many restrictions of various kinds. Because of this, it is necessary to identify as exhaustively as possible the risks that may arise and how they might affect the objectives of the project. This activity should be part of the organizational culture permeating at all levels so that it can be incorporated into the project management and carried out throughout the life cycle of the project. The idea at the end is to evaluate how well the risk management performed, i. e., if risk responses were effective and what was the influence they have on the project objectives (Hillson, 2009; Zhang, 2016; Dikmen et al., 2008; Likierman, 2007).
This was evident in a survey conducted in 2013 by KPMG Consulting Company, an international financial and taxation consultant based in Holland. It showed within its main results that risk management in the construction industry is one of the greatest challenges that companies and projects are facing, because it has prevented companies from having an adequate growth. It also pointed out that the lack of the application of risk management causes projects with delays, which end up with a high over cost and a worn relationship with customers; it calls the attention that these same results were observed a decade earlier by the publication Re-Thinking Construction (Dallas, 2008).
Then, besides what is described, it is stated in this paper that another cause of this situation is the lack of measurement of what happened during the realization of the project at its end or after to find out if the risk management was effective or not. Then, it is proposed that a methodology should be designed to achieve this purpose and that, with the information provided by the methodology, it would be possible to find out how PRM really works and obtain lessons learned for future projects.
This research aims to propose a preliminary model for measuring the performance of PRM. To achieve this, the following stages were realized:
Literature review: various databases, scientific papers, and books were consulted. An attempt was made to identify if there is any model proposed to measure the performance of PRM and what would be the relevance of being able to do it. This review allowed the identification of key PRM concepts, the establishment of the existing gaps, and the creation of a framework of reference. Development of a preliminary proposal of the performance measurement model: in this stage, risk management experts/practitioners were interviewed, being all of them persons involved in the construction industry, the insurance industry, or the academic world. Interviews were semi-structured, and main topics addressed were risk and uncertainty, risk management, and the process of risk management based on its actual application to construction projects. In the second stage, key performance variables were defined and the preliminary model for the measurement of the performance of risk management in construction projects was constructed based on the application of risk responses. Validation of the model: once the previous stage was completed, risk management experts and practitioners validated the model, the KPIs, and its application through an analysis and evaluation. The Delphi method was applied for getting a valid consensus from experts.
It was possible to observe in the literature that there are deficiencies in the application of risk management in construction projects. Moreover, there is no tangible evidence of the benefits for a company of applying risk management.
In the process of risk management, a key factor for efficient performance is the application of responses to risks. These responses, in their majority, had to be selected previously during the planning stage of the project. Still, during the execution of the project there are unidentified risks to which responses should be activated. Then, the question that arises is: did these responses work? In order to know the answer, it is essential to make a post facto evaluation once the project execution has finished.
The responses implemented are directly linked to the risks that were duly identified, analyzed, and evaluated considering the impact that the risk could cause in any or all of the project’s objectives. Hence, there is an importance to determine the factors that influence the performance of risk management.
Therefore, a series of interviews were conducted with experts on the subject and, from this point of view, conclusions were drawn on the relevant issues for a proper management of risks and the know-how about how the industry really conceives and applies risk management in construction projects. In order to carry out these interviews, experts in the field of construction who work in both the academic and professional fields were consulted.
The results of these interviews and the review of the literature show that risk identification is considered as one of the most important stages of risk management for construction projects and on which it depends on the proper completion of the following stages of this process.
In addition, the interviewed experts agreed that the process of risk management is an activity that is carried out only at the initial stages of the project, since insurance companies play an important role in the signing of the contract and in obtaining resources for projects. However, a gap exists in terms of identifying if the expected risks really occurred, if their analysis was correct, if the applied responses worked appropriately if risks occurred, and if there is an evidence that some benefits were obtained from PRM.
For this reason, using the knowledge obtained from experts and literature, a research framework was developed, including the main variables that should be considered for the measurement of the performance of PRM at the end of any project. This framework is indicated in Figure 1.
Fig. 1
Framework for measuring the performance of PRM.
At the end of the project, the performance of PRM is then measured using the KPIs associated with a combination of these variables according to the model shown in Figure 2 and taking into account the factors that were mentioned before.
Fig. 2
Proposed model for measuring the performance of PRM. PRM, project risk management.
As this is an approach to measure the performance of risk management at the end of the project, the key input variables for each factor were identified based on the importance of the role of each factor.
The variables for measuring the performance of PRM that are included in the model above are described as follows:
Performance of the risk identification process: it seeks to evaluate how effective was the identification of the risks that occurred and includes the following factors:
Risks that occurred: refers to risks that occurred during the execution of the project: Previously identified risks: the risks that occurred and were identified. Unidentified risks: these are the risks that occurred during the execution of the project but were not identified. Performance of the risk analysis and evaluation process: it seeks to evaluate how effective the analysis and evaluation of the risks was in terms of the expected impacts on the risks that occurred.
Nonresponse impacts of the risks that occurred: Expected impacts without risk response: those expected impacts of previously identified risks according to their analysis and evaluation. Real impacts of the risks with response: the real impact of the risks that occurred and to which a response was applied. It includes even those risks that occurred and where the response was to do nothing. Probability of occurrence of risks: to determine what frequency of occurrence has the risks that arise in a certain project with characteristics such as type of project, budget, term, location, and type of contract. Maybe in the first iteration, this variable is difficult to evaluate, but the idea is that through the recurrent application of this model this variable can be evaluated. Performance of the risk response plan: it seeks to evaluate how the expected responses worked in front of the risks that occurred.
Impacts that resulted with responses to the risks that occurred. Expected impact of risks with responses: once the risks have been assessed, a proposal is made of the responses to face them and how these responses will mitigate the consequences of the risks if they do occur. Real impact of the risks with responses: the real impact of the risks that occurred with the application of the planned responses. It includes even those risks that occurred and where the response was to do nothing. Profitability of risk management: it seeks to assess whether the projected contingency or expense for the response was sufficient or not to address the risks.
Benefits versus costs: Benefits: these benefits refer to the resulting impact due to the application of the responses to the risks that occurred which helped to mitigate or eliminate these risks. This is calculated with a comparison to what would have been the impact without a response. Response application costs: they refer to the costs incurred for the application of the response. Real monetary impact of risk: what was the real monetary impact of an occurring risk with a planned response. Expected monetary impact of the risk with response.
Regarding risk analysis and evaluation performance, the variables about the impact of the risk without response were identified; then, it is necessary to compare the possible difference between the previous expected impact of the risk and the real impact of the risk. With this comparison, it is possible to know how effective the analysis and evaluation of the risk was. As shown in Figure 1, these tasks are part of the project planning stage and in most cases are the only activities that are done in a rigorous way.
For the factors related to the performance of the risk response plan of PRM, the key input variables are the expected impact of the risk with the response and the real impact of the risk with the response. They will allow to know how effective the responses were, an essential part of the analysis. The application of the response obeys on the one hand to the previous risk identification and risk analysis and evaluation. On the other hand, the non-identified risks do not have a response, and in some way this might cause an overrun on the budget and the schedule. One of the common responses for these kinds of risks is an insurance or an amount of contingency considered for the project. In many cases, the contingency is not sufficient, because there are not lessons learned in the organization.
The variables related to the profitability of PRM are another important part regarding the measurement of the performance of risk management. The benefits versus the cost of the response’s application need to be clearly evidenced. The comparison between the investment for the mitigation of those risks that were or not identified and the cost of the impact with no response are relevant for the budget and are not taken into account currently at the end of a project.
The model described above has been used to design a prototype for its application. The basic architecture of this prototype is shown in Figure 3. The goal is that users will be able to interact with the prototype to obtain knowledge about lessons learned from previous projects as the risk performance information will be available from data from these projects.
Fig. 3
Basic prototype for the performance of risk management.
This paper has presented a research effort that aims to address a topic that has not been investigated largely according to what the literature indicates. The research has sought to obtain a model that correctly reflects the factors that are involved in the performance of risk management in construction projects in order to use these factors to measure this performance. The model presented is the result of a series of interviews and the first validation of experts. The next step is to structure the KPIs for addressing the project effectiveness of risk management.
This model presents a different perspective, where it is noted that risk management is an activity to be carried out at the end of the project and based on the risks that have occurred.
It is expected that the measurement of the performance of risk management in construction projects will provide construction companies with a valid and practical knowledge of the behavior of this important function and that it will allow to have valuable lessons learned to produce a substantial improvement of this process that is urgent to incorporate in construction project management.
Likewise, it is expected that this information could demonstrate the value of risk management in an industry that does not apply it systematically, suffering the consequences of this failure. The medium- and long-term objective of this research effort is to improve the performance of construction projects with better compliance of their objectives. Then, risk management could be a lever of great impact on this regard as well as in changing the paradigm that risk management is a bureaucratic, non-useful process.