Risk-based integrated performance assessment framework for public-private partnership infrastructure projects
Article Category: Research Paper
Published Online: Aug 28, 2024
Page range: 251 - 274
Received: May 11, 2024
Accepted: Jul 18, 2024
DOI: https://doi.org/10.2478/otmcj-2024-0018
Keywords
© 2024 Anam Fatima et al., published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
Public-private partnership (PPP) projects have emerged as pivotal innovative procurement models in contemporary infrastructure development, playing a transformative role in the delivery of public services across various sectors. The significance of PPP projects lies in their capacity to address the growing demands for essential infrastructure, such as health, education and transportation, particularly in the face of increasing global population trends (Soomro and Zhang 2016). These collaborative ventures involve a strategic alliance between public and private entities to finance, construct and manage public facilities, transferring them to public-sector authorities after a predefined concession period (Brinkerhoff and Brinkerhoff 2011).
However, governments face challenges in financing the expanding infrastructure needs so PPPs represent a cooperative approach wherein private entities are invited to invest, construct, own and operate public facilities, contributing to the fulfilment of critical infrastructure needs (Kang et al. 2019). These projects offer an alternative means for governments to overcome financial constraints, especially in developing nations grappling with challenges related to national debts, limited budgets and funding options (Yurdakul et al. 2022). PPP projects have demonstrated their ability to deliver higher value for money (VfM), offering advantageous returns on both cost and time invested (Yuan et al. 2012). The overarching significance of PPP projects lies in their ability to provide efficient, cost-effective solutions that bridge the gap between escalating infrastructure demands and constrained public resources (Borole 2022). By leveraging the expertise and resources of the private sector, PPP projects aim to optimise project outcomes, enhance service quality and promote sustainable development.
In the global or regional context, PPPs have become a prevalent and influential mechanism, reshaping the landscape of economic development. The global evolution of PPPs gained momentum in the 1970s and 1980s, marked by a shift towards involving private entities in infrastructure, construction and development projects (Chan et al. 2011). This paradigm shift was further fuelled by the Public Management Movement, emphasising the competition between private and public sectors in infrastructure development. In the past two decades, utilisation of PPPs has facilitated governments in enhancing the delivery of infrastructure projects, allowing for improved provision of public services (Pongsiri 2002). Globally, PPPs have proven to be instrumental in addressing the limitations of traditional government financing and delivery models. Their prevalence is particularly notable in mature markets such as Australia and the UK, where sophisticated PPP frameworks have been established, contributing to successful project delivery (Hodge 2004; Liang and Jia 2018; Ward et al. 2019; Nguyen et al. 2020; Berisha et al. 2022). Normative literature has documented numerous successes and occasional failures in PPPs. However, contemporary discourse surrounding PPPs has transcended ideological debates regarding their merits and drawbacks. Instead, the discussion now centres on structuring PPPs that aligns with public policy objectives (Hodge 2004; Yong 2010; Regan et al. 2011a, 2011b; Berisha et al. 2022). To harness the potential advantages offered by PPPs, they need to be structured to enhance performance while distributing costs and risks between the public and private sectors (Yong 2010). However, despite their experience in implementing PPPs, inadequate performance measures (PMs) have been pinpointed as a contributing factor to the challenges encountered during the delivery of projects such as the Latrobe Regional Hospital and Deer Park Women’s Prison in Victoria, Australia, and Ashfield Prison in the United Kingdom (VAGO 2002; House of Commons Committee of Public Accounts 2003; Roth 2004). According to Yuan et al. (2009) the absence of effective PMs in PPPs acts as a trigger for producing below-optimum service quality of infrastructure. However, many PPP projects have not undergone a comprehensive form of
Implementing a robust performance assessment framework for PPP projects yields multifaceted benefits, enhancing accountability, risk management and overall project success (Yong 2010; Szemere et al. 2021). Effective risk management is another critical advantage of an integrated performance assessment framework (IPAF). Identifying, analysing and allocating risks play pivotal roles in PPP endeavours, especially given the limited obligation of private investors in design and construction, with the public sector bearing the brunt of financial and operational uncertainties (Nawaz et al. 2019; Sarvari et al. 2019). Extensive research has explored PPP risks, offering valuable perspectives on identifying and distributing these risks effectively (Wang et al. 2000a, 2000b; Grimsey and Lewis 2002; Thomas et al. 2003; Bing et al. 2005; Xu et al. 2010). By assessing and identifying risks systematically throughout the project lifecycle, stakeholders can implement mitigation strategies proactively (Valipour et al. 2018). This proactive approach minimises the likelihood of project disruptions, financial setbacks and legal disputes, thereby safeguarding the interests of both public and private participants (Floricel and Miller 2001).
Numerous researchers have outlined key elements vital for the successful implementation of PPP projects, emphasising the pivotal role of PMs (Hodge and Greve 2007; Yuan et al. 2009). According to Yong, stakeholders must prioritise performance measurement issues during project development to realise PPP benefits (Yong 2010). Performance measurement is crucial for achieving business success whether at the corporate or project level (Love and Holt 2000; Kagioglou et al. 2001). Monitoring and evaluating performance throughout the PPP life cycle are fundamental activities for contract and project management professionals, constituting an essential aspect of PPP policy in various countries (Chinyio and Gameson 2009; EIB 2012). Inadequate performance measurement in PPPs often leads to diminished infrastructure service quality (Liang and Wang 2019). However, despite the critical importance of performance measurement to project success, comprehensive assessments of many PPPs are still lacking (Regan et al. 2011b; Liu et al. 2015).
Given this context, this study reviews the literature on distinctive features and requisite evaluations of PPP projects, along with the broader domain of performance measurement in construction, it underscores the significance of achieving a balance between performance measurement and management processes. The study highlights the need for a comprehensive framework that not only assesses project performance but also integrates robust risk management strategies to mitigate uncertainties and enhance outcomes (Franco-Santos et al. 2012; Bititci 2015; Smith and Bititci 2017). It also aims to present an IPAF for evaluating PPP infrastructure projects, identifying clear and measurable critical risk factors associated with PPP projects and proposing effective mitigation techniques. This inclusive approach aims to evaluate the performance of PPP project management by furnishing stakeholders with insightful information on project performance and highlighting areas for improvement, fostering a more robust and nuanced understanding of project success. Furthermore, the study aims to contribute to the existing literature by exploring how the uncertain and volatile operating environment affects performance measurement and management in PPP projects (Nudurupati et al. 2021). By prioritising risk management within the Integrated assessment framework, the study seeks to offer insights into effectively managing the inherent complexities of PPP infrastructure projects (Smith and Bititci 2017; Bourne et al. 2018; Sardi et al. 2020).
Performance measurement in PPP projects encompasses efficiency, effectiveness and VfM (Lebas 1995; Akintoye et al. 2003; Grimsey and Lewis 2005; Solomon and Young 2007). VfM aims to balance cost and quality to meet user requirements (Office of Government Commerce 2002). While conventional assessments prioritise cost, incorporating time and quality dimensions allows for a more comprehensive evaluation (Henjewele et al. 2014). Neglecting stakeholder satisfaction and project phases may result in incomplete assessments (Liu et al. 2022). Various dimensions, including project quality, cost, duration, safety and environmental sustainability, significantly influence PPP performance (Lim and Mohamed 1999; Kumaraswamy and Zhang 2001; Yin 2009; Jin and Zhang 2011; Liu et al. 2015; Fellows and Liu 2021). Tailored assessment models acknowledge the multifaceted nature of PPP projects and advocate for nuanced approaches (Kumaraswamy and Zhang 2001; Cong and Ma 2018; Liang and Wang 2019; Esposito and Dicorato 2020). The incorporation of sustainable development goals (SDGs) underscores the increasing emphasis on sustainability within PPP initiatives.
Developing a performance assessment framework for PPP projects is imperative for several reasons. First, existing literature showcases various frameworks and models for evaluating PPP project performance, highlighting the diverse dimensions and criteria involved. However, their success hinges on effective performance assessment methodologies that account for inherent risks. Risk-based performance assessment in PPP projects is essential for identifying, managing and mitigating risks throughout the project lifecycle, ensuring optimal outcomes and value for stakeholders. For this purpose, this study aims to
Develop a systematic performance assessment framework for PPP projects, prioritising risk management in developing country contexts. Integrate key PMs and risk mitigation strategies across the feasibility, execution and operation and maintenance phases of PPP projects. Enable informed decision-making and enhance project outcomes through the structured application of the developed framework.
Considering the studies and framework examples presented above, given Pakistan’s status as a developing country, the need to establish a robust performance assessment framework for PPP projects takes on heightened significance. As Pakistan endeavours to accelerate its economic and infrastructure development, effective evaluation mechanisms become indispensable tools for ensuring accountability, transparency and efficient resource utilisation within PPP initiatives. Recent studies in Pakistan, such as (Ullah et al. 2016, 2018; Noor and Khalfan 2017; Javid 2019; Nawaz et al. 2019; Ahmad et al. 2022), underscore the importance of adapting global best practices to the local context and addressing the unique challenges faced by PPP projects in Pakistan. By forging an IPAF tailored to the unique socio-economic context of Pakistan, stakeholders can effectively monitor project progress, identify areas for improvement and mitigate risks inherent in PPP ventures. This framework not only fosters greater accountability among public and private entities but also facilitates evidence-based decision-making to optimise resource allocation and maximise the impact of infrastructure investments.
Moreover, the exploration of current gaps in the literature and the pursuit of future research avenues hold immense potential in advancing the understanding and practice of performance assessment in PPP projects within the Pakistani context. By addressing these gaps, researchers can contribute valuable insights that inform policy formulation, refine project management practices and ultimately enhance the efficacy and sustainability of PPP endeavours in driving socio-economic development across Pakistan. In essence, developing a robust performance assessment framework for PPP projects in Pakistan represents a pivotal step towards realising the country’s developmental aspirations. Through concerted efforts to bridge existing knowledge gaps and innovate within the scope of performance assessment, Pakistan can chart a path towards more inclusive, resilient and prosperous infrastructure development outcomes.
The study aims to provide valuable insights into the performance assessment of PPP projects in Pakistan through a rigorous research methodology, contributing to the advancement of knowledge in the field. The research methodology for this study is structured around a comprehensive understanding of research principles and methods (Ahmad et al. 2022). The research engages with various types of methodologies, with a primary focus on mixed-methods research incorporating qualitative and quantitative approaches as shown in Figure 1. The qualitative methodology is considered suitable for this research investigation as it enables thorough exploration and data collection to establish the grounded theory. This theory could certainly be confirmed through a quantitative approach using a larger dataset (Noor and Khalfan 2017). The methodology draws from the work of Jin and Zhang (2011) to identify qualitative PMs, Almarri and Abuhijleh (2017) for expert review, and data for the focus group interviews were gathered from the studies of Yin (2009) and Zhang et al. (2015), project documentation review from Saeed et al. (2018), and site observations from Trangkanont and Charoenngam (2014), methodologies previously employed in studies concerning performance evaluation frameworks for PPP projects.
Theoretical model of research methodology.
The first stage of the research commenced with a comprehensive literature review aimed at identifying risk factors, key performance indicators (KPIs) and their corresponding PMs. Various academic sources, including journals, conferences and books, as well as non-academic materials like institutional reports and archives, were extensively consulted in this stage. Initially, KPIs were extracted from relevant studies conducted in developing countries (Liu et al. 2016; Noor and Khalfan 2017; Mazher et al. 2018; Ullah et al. 2018; Soomro et al. 2020; Ahmad et al. 2022). Subsequently, a meticulous review of 91 pertinent articles was conducted to augment the theoretical foundation. From these efforts, a total of 99 Risk Factors and 114 PMs were distilled from the literature, highlighting the extensive spectrum of PPP performance within the framework of the iron triangle concept (Liu et al. 2016). Questionnaires are designed based on insights from the literature review and undergo iterative refinement. A two-phase approach is employed, starting with a pilot survey to refine the questionnaire design to incorporate additional PMs that were not previously integrated, thereby enhancing the comprehensiveness of the framework, The pilot survey was digitally administered, offering considerable savings in time and costs when compared to traditional paper-based methods (Van Selm and Jankowski 2006) followed by the development of a main questionnaire for focussed interviews, where participants rated the significance of responses on a five-point scale, with ‘5’ indicating extreme importance and ‘1’ denoting insignificance. The relative importance of identified PMs and Risk factors were ranked using the relative importance index (RII) analysis method. The RII ranges from 0 to 1 and is used to rank factors based on their importance, with higher values indicating greater importance. An RII closer to 1 means that the factor is considered very important, while an RII closer to 0 indicates that the factor is less important. To calculate the relative importance of each item, the following formula was used (Masood et al. 2021)
W – weight of each factor (i.e. 1–5) A – highest weight of factor (i.e. 5) N – total number of participants.
The higher the likelihood of occurrence, the higher will be the RII Rating. After finding the RII, all the risk factors and PMs were ranked. The frequency of each point on the Likert scale for every risk factor and PM was determined using Statistical Package for Social Sciences (SPSS).
The sample size selection process for this qualitative research is contextual, based on the availability of participants and their level of experience in PPP projects. Baker and Edwards (2012) argue that there is no specific rule regarding the sample size for the number of interviews required in qualitative research, whereas Cresswell (2013) suggests a general rule of thumb for achieving data saturation. After conducting 10 interviews, it became evident that several risk and performance parameters presented similar responses from PPP experts, indicating a saturation point where no new information was gained. Consequently, only 10 interviews were conducted for this research.
The data collection methods involve an initial online questionnaire phase followed by focussed interviews, employing the triangulation method to ensure comprehensive data gathering. Triangulation enhances the validity and reliability of findings by integrating data from multiple sources and methods (Ahmad et al. 2022). To determine the internal consistency for sample reliability, Cronbach’s alpha was calculated using the SPSS tool and found to be 0.896 which is >0.7 indicating that data is reliable for further analysis (Tavakol and Dennick 2011).
The second stage involves the analysis of gathered data, the focus of data collection centred around two main components: risk factors and PMs and collation specifically involve mapping risks to the PMs they directly influence (Noor and Khalfan 2017). Regarding risk factors, the categorisation process encompasses two main steps. First, an extensive literature review identifies and categorises risks into eleven distinct categories, including financial, contractual, political, construction and environmental risks, among others. Second, these risks are further segmented based on project phases, including feasibility, construction and operation and maintenance phases. A part of the questionnaire was used to validate and finalise the identified risk categories. The analysis of gathered data includes conducting the RII to measure the significance of both risk factors and PMs (Masood et al. 2021). Table 1 displays the 43 shortlisted risk factors along with their coding chosen for the RII test based on their relevance and occurrence. For the PMs, initially, a literature review compiles relevant PMs for PPP projects. A subsequent survey refines this list through interviews and analysis. The finalised set of 16 PMs out of 114, aligned with the key indicators, is detailed in Table 2, reflecting associations with cost, time and quality considerations. This iterative data collection and analysis process ensures the validity and reliability of the research findings.
Categorised critical risk factors
| Stakeholder Risks | Reference | |
|---|---|---|
| Cultural differences between stakeholders | Ezatabadi et al. (2018) | |
| Communication Issues | Rasheed et al. (2022) | |
| Lack of leadership, authority, organisation and coordination | Ezatabadi et al. (2018) | |
| Lack of commitment by a private party | Abd Karim (2011) | |
| CP1 | Improper project feasibility study and planning | Sy et al. (2016) |
| CP2 | Incomplete Design | Rasheed et al. (2022) |
| CP3 | Improper change order approval process | Chan et al. (2011) |
| CP4 | Poor definition of scope and change in scope | Rasheed et al. (2022) |
| CP5 | Inappropriate project specification | Wang et al. (2020a) |
| CP6 | Inadequate Site Investigation | Chan et al. (2015) |
| CP7 | Losses to equipment and properties of contractors | |
| CP8 | Documentation Errors | Xu et al. (2015) |
| CP9 | Accidents on sites and Injuries | Yuan et al. (2018) |
| CP10 | Unavailability of labour and material | Xu et al. (2010) |
| CP11 | Poor Workmanship | Rasheed et al. (2022) |
| CP12 | Third-Party Delay and Violation | Xu et al. (2010) |
| OP1 | Low Operating Productivity | Abd Karim (2011) |
| OP2 | Lack of Public Acceptance of PPP Project | Zou et al. (2008) |
| OP3 | Unsuitable Concession Period | |
| OP4 | Unpredictable Revenue Generation | |
| OP5 | Poor O&M of Project | Qin et al. (2019) |
| OP6 | Lack of Compliance of Contract | Rasheed et al. (2022) |
| F1 | Financers Unwilling to take risks | Abd Karim (2011) |
| F2 | Improper Risk Allocation | Koike and Hofert (2020) |
| F3 | Delay in financial closure | Abd Karim (2011) |
| F4 | Delay in payment of annuity | |
| F5 | Delay in funding | Wang et al. (2020b) |
| F6 | Concessionaire Change | Xu et al. (2010) |
| F7 | Change in project cost | Zou et al. (2008), Omoregie Aghimien et al. (2017) |
| F8 | Minimum Revenue Guarantee | Zou et al. (2008) |
| F9 | Inflation and inaccurate estimate of market demand | Zou et al. (2008), Xu et al. (2010) |
| F10 | Inadequate Insurance Coverage | Chan et al. (2015) |
| F11 | Improper Financial Model | Abd Karim (2011) |
| L1 | Legal Proceedings | Soomro and Zhang (2015) |
| L2 | Contractual changes and ambiguities | |
| L3 | Lack of federal, state and local permits | Abd Karim (2011) |
| G1 | Improper Regulatory Framework | Abd Karim (2011) |
| G2 | Inconsistency in governmental policies and regulations | |
| G3 | Lack of political support | Xu et al. (2010) |
| G4 | Change of Government | Zou et al. (2007) |
| EG1 | Act of God/Force Majeure Risks | Xu et al. (2010) |
| EG2 | Corruption | Zou et al. (2007) |
| EG3 | Land Acquisition | Xu et al. (2010) |
PPP, public-private partnership.
KPIs and PMs in PPP projects.
| Sr. No. | KPI | Code | PM | Research Question | Measurement | Reference |
|---|---|---|---|---|---|---|
| 1. | KPI1 Cost | PM-1 | Construction Cost Variance | What is the disparity between the budgeted and actual construction expenses for the project? | Rupees | Soomro et al. (2020), Ullah et al. (2018) and Yuan et al. (2012) |
| 2. | PM-2 | Construction Cost | Was the construction cost optimised effectively? | Yes/No | Heale and Forbes (2013), Mazher et al. (2018) and Ying et al. (2024) | |
| 3. | PM-3 | Viability of the Financial Model | Is the financial model considered to be feasible? | Yes/No | Guo et al. (2021), Jin et al. (2020) and Ke (2014) | |
| 4. | PM-4 | VfM | What is the VfM for the project? | Rupees | Buyukyoran and Gundes (2018), Ingle et al. (2021) and Nguyen et al. (2018) | |
| 5. | KPI2 Time | PM-5 | Construction Time Variance | What distinguishes the allocated construction time from the actual construction time? | Days | Attarzadeh et al. (2017), Matraeva et al. (2016) and Yuan et al. (2012) |
| 6. | PM-6 | Project Completion and Delays | Was there any delay in completing the project? | Yes/No | Guo et al. (2021), Mladenovic et al. (2013) | |
| 7. | PM-7 | Concession Period and Recovery | Was the concession period utilised optimally? | Yes/No | Malek and Gundaliya (2021), Yong Kim and Thuc (2021) | |
| 8. | PM-8 | Defect Liability Period | What was the duration of the defect liability period for this project? | Years | Pilot Study | |
| 9. | KPI3 Quality | PM-9 | Quality Specified vs Actual Quality | Does the actual quality meet the listed standards? | Yes/No | Ahmadabadi and Heravi (2019), Guo et al. (2021), Osei-Kyei and Chan (2017) and Villalba-Romero and Liyanage (2016) |
| 10. | PM-10 | Health and Safety | Were the on-site health and safety provisions sufficiently provided? | Yes/No | Bao et al. (2018), Buertey and Asare (2014) | |
| 11. | PM-11 | Defects and Problems | Were maintenance duties efficiently executed? | Yes/No | Mohamad et al. (2018), Radujković et al. (2010), Villalba-Romero et al. (2015) and Wang et al. (2020b) | |
| 12. | PM-12 | Transparency | Were project issues and problems transparently communicated throughout? | Yes/No | Pilot Study | |
| 13. | PM-13 | Effectiveness | Was the project compliant with the specified standards and objectives? | Yes/No | Pilot Study | |
| 14. | PM-14 | Efficiency | Was the project executed efficiently? | Yes/No | Cong and Ma (2018) | |
| 15. | PM-15 | Communication | Was the communication among stakeholders efficient and effective? | Yes/No | Pilot Study | |
| 16. | PM-16 | Procurement | Were quality products and services procured as required? | Yes/No | Pilot Study |
KPI, key performance indicator; PM, performance measure; PPP, public-private partnership; VfM, value for money.
In the third stage, the study formulates an IPAF for PPP projects based on 3 KPIs and 16 PMs derived from literature reviews and interviews, addressing the top 10 critical risk factors across various categories including stakeholder, construction project, financial, legal, governmental and environmental risks.
In stage four, the contents of the theoretical IPAF were reviewed by four PPP experts to assess their appropriateness and relevance to PPP projects across the region. This review functioned as an initial exploratory study to ensure alignment with the project’s nature and the context of a developing country, precisely focussing on privately financed public sector infrastructure projects in Pakistan. The experts were selected based on their extensive involvement in PPP projects and their experience across various stages of project development. The panel, consisting of four experts, was carefully assembled to ensure a comprehensive evaluation process, each member brings distinct expertise to the table, ensuring that the number of participants is justified based on their relevant experience for the focussed group method (Masood et al. 2023). The first expert held the position of Deputy Director of Contracts (Works) at Punjab Road Road Authority (PRRA). The second expert served as Deputy Director of Engineering (Works) at PRRA. The third expert functioned as a Coordination Manager at LAFCO (Lahore – Sheikhupura – Faisalabad Dual Carriageway) and the fourth expert was the Assistant Manager of Evolutions at the Punjab PPP Authority. All experts possessed relevant qualifications in construction and brought over 10 years of experience in PPP projects, particularly related to highway infrastructure projects.
The consistency of expert responses was ensured through several steps. First, pilot testing of the questionnaire was conducted to resolve any issues and to ensure clarity, subsequently, focus group interviews were carried out to validate and clarify responses further (Yeong et al. 2018). By training participants, maintaining anonymity and randomising response options, we aimed to minimise bias and ensure reliable data (Hallowell and Gambatese 2010). Finally, thorough data analysis helped to identify and address any inconsistencies, thereby enhancing the credibility of our study.
The results highlight the nuanced approach required for performance measurement in construction projects, emphasising dimensions like cost, time and quality. Synthesising insights from literature reviews and interviews, the study uncovers the complexities of assessing project performance across diverse contexts. Through critical examination, the research contributes to ongoing discourse on effective performance measurement strategies.
The study utilised SPSS for data analysis for processing triangulated data collected through interviews (Lemon and Hayes 2020). Data, structured with a Likert scale, transitioned from qualitative to quantitative during interviews. SPSS performed frequency analysis, and spreadsheets computed the RII for risks, to find the top 10 risks, present in the PPP Industry of Pakistan out of 43 risks. Using the RII found for every risk, the bar chart in Figure 2 was constructed.
Analysis of top 10 risks in the PPP industry of Pakistan using RII technique. PPP, public-private partnership; RII, relative importance index.
Table 3 outlines the top 10 risks as determined by the RII, shedding light on critical factors influencing project outcomes within the study’s context. These risks span various categories, each identified by a unique ID. Financial Risk (F-9) emerges as the foremost concern, encompassing challenges related to inflation and inaccurate market demand estimates, which could significantly impact project finances. Environmental and General Risk (EG-1) follows closely, representing unpredictable events like Acts of God or Force Majeure, posing substantial threats to project continuity and progress. Operational Risk (OP-4) highlights the unpredictability of revenue generation, underscoring the importance of financial planning and sustainability measures. Meanwhile, Construction Project Risk (CP-1) underscores the significance of thorough feasibility studies and planning to mitigate potential challenges arising from improper project assessments. Governmental Risk (G-2) reflects the instability caused by inconsistencies in governmental policies and regulations, which could introduce uncertainty and hinder project execution. Moreover, Operational Risk (OP-3) addresses the suitability of concession periods critical for ensuring project viability and operational efficiency. Construction Project Risk (CP-12) identifies third-party delays and violations as potential impediments to project timelines and contractual agreements. Legal Risk (L-2) emphasises the importance of clear contractual terms to avoid legal complications and disputes stemming from contractual changes and ambiguities. Financial Risk (F-7) relates to fluctuations in project costs, necessitating budgetary flexibility and contingency planning to address unforeseen expenses. Finally, Construction Project Risk (CP-8) underscores the need for accurate documentation to prevent errors that could undermine project integrity and performance. Overall, these risks underscore the multifaceted challenges inherent in PPP projects, emphasising the importance of proactive risk management strategies to ensure successful project delivery and mitigate adverse impacts on project outcomes.
Top 10 risk ranking by RII
| Rank | Risk Category | Risks | RII Value |
|---|---|---|---|
| 1 | F-9 Financial Risk | Inflation and inaccurate estimate of market demand | 0.8 |
| 2 | EG-1 Environmental and General Risk | Act of God/Force Majeure Risks | 0.78 |
| 3 | OP-4 Operational Risk | Unpredictable Revenue Generation | 0.71 |
| 4 | CP-1 Construction Project Risk | Improper project feasibility study and planning | 0.7 |
| 5 | G-2 Governmental Risk | Inconsistency in governmental policies and regulations | 0.68 |
| 6 | OP-3 Operational Risk | Unsuitable Concession Period | 0.643 |
| 7 | CP-12 Construction Project Risk | Third-Party Delay and Violation | 0.642 |
| 8 | L-2 Legal Risk | Contractual changes and ambiguities | 0.64 |
| 9 | F-7 Financial Risk | Change in project cost | 0.581 |
| 10 | CP-8 Construction Project Risk | Documentation Errors | 0.58 |
RII, relative importance index.
Furthermore, a total of 16 PMs were considered for 3 KPIs: cost, time and quality. Table 4 presents the ranking of PMs based on their RII, offering insights into key areas of focus for assessing project performance. The table categorises PMs according to their respective KPIs and assigns unique IDs to each measure. The top three PMs are respectively for KPI time, cost and quality. Under the KPI1, PM-6 ranked first due to its importance and it highlights Project Completion and Delays as a crucial metric for evaluating project timelines and adherence to schedules. Within the KPI2 category, PM-3 ranked second, assesses the Viability of the Financial Model, emphasising the importance of financial sustainability and feasibility. KPI3-related PMs include PM-9 which is ranked third, it evaluates Quality Specified vs Actual Quality, indicating the alignment between project specifications and actual outcomes. Lastly PM-15 (Communication) and PM-12 (Transparency) are ranked the lowest as it underscores the importance of safety protocols, effective communication and transparency in project operations. Overall, the table provides a comprehensive overview of performance metrics essential for evaluating PPP project outcomes and ensuring project success. All of these PMs were analysed through the RII technique as shown in Table 4. RII values ranked risks and PMs facilitated during the formulation of a PPP project performance assessment framework.
PMs ranking by RII
| Rank | KPIs | ID | Performance Measure (PMs) | RII Value |
|---|---|---|---|---|
| 1 | Time | PM-6 | Project Completion and Delays | 0.98 |
| 2 | Cost | PM-3 | Viability of the Financial Model | 0.921 |
| 3 | Quality | PM-9 | Quality Specified vs Actual Quality | 0.92 |
| 4 | Cost | PM-2 | Construction Cost | 0.902 |
| 5 | Cost | PM-5 | Construction Time Variance | 0.901 |
| 6 | Quality | PM-14 | Efficiency | 0.90 |
| 7 | Quality | PM-13 | Effectiveness | 0.88 |
| 8 | Quality | PM-16 | Procurement | 0.86 |
| 9 | Cost | PM-1 | Construction Cost Variance | 0.843 |
| 10 | Quality | PM-11 | Defects and Problems | 0.841 |
| 11 | Time | PM-7 | Concession Period and Recovery | 0.84 |
| 12 | Cost | PM-4 | VfM | 0.822 |
| 13 | Time | PM-8 | Defect Liability | 0.82 |
| 14 | Quality | PM-10 | Health and Safety | 0.8 |
| 15 | Quality | PM-15 | Communication | 0.68 |
| 16 | Quality | PM-12 | Transparency | 0.66 |
KPI, key performance indicator; PM, performance measure; RII, relative importance index; VfM, value for money.
Key findings included top risk identification and PM ranking, reflecting extensive respondent experience in PPP projects. The analysis pinpointed prominent risk categories and PM importance, laying the groundwork for informed decision-making and risk mitigation strategies.
The performance measurement framework was developed based on the findings already discussed, categorising data according to its significance in PPP project performance measurement. Risks and PMs were organised based on their RII. The framework was structured across three levels: Level 1 – Risk Integration, Level 2 – Risk factors mapping with PMs and Level 3 – CPAF, integration of Risk with PMs in PPP Project Phases.
At Level 1, the focus lies on understanding the relationship between risks. Risks can be either independent or dependent on each other, impacting mitigation strategies.
Based on the approach outlined by Utne et al. (2011), the risk interdependency diagram was developed, as depicted in Figure 3. This diagram illustrates the relationship between the top 10 risks, highlighting both independent and dependent risks.
Risk interdependency diagram of top 10 risks.
A Risk Interdependency Diagram is a visual representation that illustrates the relationships between different risks within a system or project, it highlights how various risks interact with each other, showing dependencies and potential impacts (Tamimi 1989). This diagram helps stakeholders understand the complexity of risk scenarios and identify key areas for mitigation and management. The dotted line shows the risks that are independent of each other, while the solid line with an arrowhead shows the independent risk on its tail side and the dependent risk on the arrow side. The dependent risk is affected by the independent risks present at the tail end and thus can occur because of the independent risk. The mitigation strategy for the dependent risk can be the same as that applied to the risk that causes it (Tamimi 1989).
Figure 3 provides a visual representation of the interdependency among the top 10 risks, distinguishing between independent and dependent risks. Notably, OP-3 is reliant on a single risk, CP-1. CP-12 exhibits dependence on two risks, namely L-2 and G-2. F-7 demonstrates dependency on three risks: CP-12, F-9 and EG-1 while remaining independent of OP-4. Conversely, OP-4 depends on EG-1, OP-3, CP-1 and F-9, yet remains independent of F-7. Finally, CP-8, G-2, EG-1, L-2, F-9 and CP-1 are autonomous from all other risks, indicating their independent nature within the framework.
Level 2 explores the connection between risks and the PMs they influence. As depicted in Appendix A, a detailed breakdown of the PMs aligned with KPIs and their associated risk factors along with integration solutions. Each KPI is accompanied by a set of PMs, and each PM is linked with specific risk factors that may impact its performance. For instance, under KPI1 (Cost), PM-1 (Construction Cost Variance) is influenced by risks such as F-9 (Inflation and inaccurate estimate of market demand) and EG-1 (Act of God/Force Majeure). KPI2 (Time) includes PM-5 (Construction Time Variance), which is affected by risks including EG-1 (Act of God/Force Majeure) and CP-12 (Third-Party Delay and Violation). Similarly, KPI3 (Quality) includes PM-9 (Quality Specified vs Actual Quality), which is affected by risks including EG-1 (Act of God/Force Majeure) and CP-1 (Improper project feasibility study and planning).
Appendix A not only identifies the risks associated with specific KPIs and PMs but also provides comprehensive solutions for mitigating these risks, the solutions were extracted from literature and expert opinion. These solutions include conducting accurate market research, which allows project managers to better understand market dynamics and anticipate potential fluctuations in costs or demand. Furthermore, transferring risk through insurance coverage provides a safety net against unforeseen events such as Act of God occurrences, ensuring financial stability and minimising losses. Enforcing stringent safety measures on-site is another critical aspect highlighted in the appendix, as it reduces the likelihood of accidents and ensures compliance with regulatory standards. By implementing these suggested mitigation measures, project managers can significantly enhance their ability to execute projects with reduced risk exposure, ultimately contributing to successful project outcomes and stakeholder satisfaction.
Level 3 of the research links risks and PMs with PPP project phases, offering a comprehensive framework for performance measurement as shown in Appendix A. Mitigation strategies are recommended for performance issues encountered across project stages. An influence diagram for each KPI (Cost, Time and Quality) illustrates their impact on project phases and suggests risk management techniques to mitigate negative effects as shown in Figure 4, and the details of each risk integrated with respective PMs are explained in Appendix A.
Influence diagram illustrating the impact of risks on PMs in PPP projects (integrating iron triangle). (A) Cost influence diagram, (B) Time influence diagram and (C) Quality influence diagram (right). PM, performance measure; PPP, public-private partnership.
The conclusion of the study involves aligning risks and PMs with the various phases of PPP projects, resulting in an integrated performance measurement framework. This stage offers a structured approach to assess PPP project performance concerning risks and their management. Additionally, it provides recommendations for mitigating performance issues encountered throughout different project stages. The final framework represents when each PM should be evaluated within the project phases and advocates applying risk management strategies to address any identified issues. The phases considered in a PPP project include Phase-I (Inception), Phase-II (Feasibility, Planning and Design), Phase-III (Tendering and Bidding/Procurement), Phase-IV (Execution) and Phase-V (Operation and Maintenance). The framework provides recommendations for addressing performance issues encountered at various stages of PPP projects. For instance, during Phase II feasibility studies, attention should be given to PM-1, PM-3, PM-4, PM-9, PM-13 and PM-14. In Phase III, which involves tendering and bidding, PM-12 and PM-15 are particularly relevant. In Phase IV, encompassing execution, the focus should be on PM-10, PM-12, PM-15 and PM-16. Finally, during Phase V, the operation and maintenance phase, which is crucial, attention should be directed towards PM-2, PM-3, PM-4, PM-7, PM-9, PM-11, PM-13 and PM-14. This systematic approach ensures that performance issues are adequately addressed throughout the lifecycle of PPP projects. This multi-tiered framework provides a structured approach to evaluate and enhance PPP project performance, offering valuable insights for project management and stakeholders. With detailed analysis and integration of risks and PMs, the framework facilitates informed decision-making and effective risk mitigation strategies throughout the project lifecycle.
The relationships between risks, PMs and PPP project phases were developed from literature reviews and expert insights. The IPAF was informed by existing research and validated through input from experts in the field. Similarly, the influence diagrams for each KPI (Cost, Time and Quality) in Figure 4 were developed based on both literature findings and expert opinion to illustrate their impact on project phases and suggest risk management techniques.
A cost influence diagram for KPI1 was constructed to assess how risks impact various PMs across different phases of a PPP project, with Figure 4 elucidating these relationships. For instance, PM-1 focusses on Construction Cost Variance, and evaluates deviations between estimated and actual costs, crucial for financial performance evaluation, especially during the pre-feasibility, feasibility and execution stages. Among the top risks, F-9 (Inflation and Inaccurate Estimation of Market Demand) and EG-1 (Act of God/Force Majeure) significantly influence Construction Cost Variance, with suggested mitigation strategies including accurate estimation and risk transfer. Similarly, PM-2, addressing overall Construction Cost, is directly affected by risks like EG-1 and F-7 (Change in Project Cost), with mitigation strategies involving risk transfer and meticulous feasibility studies. PM-3, evaluating the Viability of the Financial Model, encounters risks such as OP-4 (Unpredictable Revenue Generation) and CP-1 (Improper Project Feasibility Studies and Planning), with mitigation strategies including revenue scenario planning and precise feasibility studies. Finally, PM-4, assessing VfM, faces similar risks, with strategies focussing on revenue estimation accuracy and rigorous feasibility studies. Implementing these strategies ensures optimal project performance and quality, maintaining a balance between benefits and costs across project stages, while addressing changing social and economic contexts.
A time influence diagram for KPI2 was developed to assess how risks affect different performance metrics across various phases of a PPP project, as depicted in Figure 4, with a focus on evaluating Time-related KPIs. Most time-related issues occur during the project’s execution phase, while the efficiency of the concession period is scrutinised during the operations phase. Primarily, the execution phase focusses on assessing PMs related to time, notably PM-5 Construction Time Variance, influenced by risks such as EG-1 Force Majeure, CP-12 Third Party Delays and Violation and G-2 Inconsistency in Governmental Policies and Regulations. These risks, occurring during execution, necessitate robust mitigation strategies. For instance, the transfer of Force Majeure risks to third parties like insurance companies mitigates delays, while negotiations mitigate third-party violations. Moreover, consistent governmental policies and clear contractual terms help avert delays caused by policy inconsistencies and contractual ambiguities. In the context of project completion and delays PM-6, risks including third-party delays, Force Majeure events and governmental policy inconsistencies impact project schedules and require similar mitigation efforts. Notably, addressing land acquisition challenges through compensation and pre-planning mitigates delays inherent to PPP projects. Additionally, during the concession period PM-7, risks like unsuitable concession periods, third-party delays and Force Majeure events require proactive measures such as clear communication, risk assessment and transferring risks to third parties to maintain project timelines. Finally, in managing the defect liability period PM-8, strategies like risk transfer, avoidance of policy inconsistencies and mitigation of third-party delays ensure efficient defect rectification within the stipulated timeframe, safeguarding project schedules and performance metrics across all phases.
A quality influence diagram for KPI3 has been devised to assess the impact of risks on various PMs throughout different phases of a PPP project, as depicted in Figure 4. PM-9, focussing on Quality Specified vs Actual Quality, emphasises the alignment between contractual standards and actual project outcomes, crucial for project success and stakeholder satisfaction. Misalignment between these two can jeopardise project success, stakeholder satisfaction and reputation. Risks such as CP-1 Improper Planning and EG-1 Act of God/Force Majeure can impede quality attainment, necessitating meticulous monitoring and control procedures during the operation phase to ensure adherence to set criteria. Mitigation strategies involve avoiding poor planning through thorough risk assessment and transferring force majeure risks to external parties. Another vital parameter is PM-10 Health and Safety, which highlights the importance of stringent adherence to health and safety standards throughout the execution phase to prevent accidents and injuries CP-9, thereby enhancing project productivity and reputation. The risk of on-site accidents and injuries underscores the need for robust safety protocols and training. An avoidance strategy involving strict safety procedures, training and routine inspections is recommended to mitigate this risk and maintain project quality. Furthermore, PM-11 Defects and Problems emphasises the significance of promptly identifying and rectifying flaws to minimise their negative impact on project performance and quality. Risks like OP-5 Ineffective operation and maintenance procedures pose a risk to quality, necessitating robust planning and implementation of O&M protocols. Avoidance strategies involving strong contractual requirements and qualified service providers can mitigate this risk and ensure ongoing quality and durability. PM-12 Transparency is critical for fostering stakeholder trust and accountability, thus improving project coordination and decision-making. Risks like CP-8 Documentation errors and communication issues can undermine transparency, highlighting the need for standardised templates, quality control measures and proactive communication strategies. Mitigation involves avoiding errors through standardised processes and mitigating communication issues through open dialogue and efficient project management techniques. PM-13, evaluating Effectiveness, underscores the importance of mitigating risks such as EG-1 Act of God/Force Majeure and CP-1 Improper Project Feasibility Study and Planning through early consideration and mitigation planning. PM-14, concerning Efficiency, emphasises cost-effectiveness and resource management, with risks mitigated through strategies like resilient design and thorough risk assessment. PM-15, addressing Communication, emphasises clarity and openness, with risks like CP-8 Documentation Error and SH-2 Communication Issues managed through standardised documentation and proactive communication strategies. Finally, PM-16, focussing on Procurement, highlights fair selection and VfM, with risks like F-9 Inflation and EG-1 Act of God/Force Majeure managed through acceptance and transfer strategies, ensuring quality standards are upheld throughout the project lifecycle.
In the final stage of the research, the theoretical framework was scrutinised by four experts to ensure its appropriateness and alignment with the project context, especially considering the unique nature of public projects in Pakistan. The experts’ extensive experience in PPP projects enabled them to assess the framework’s relevance and applicability effectively.
The experts also meticulously evaluated the framework’s alignment with the 2017 PPP Law, ensuring coherence and consistency with established legal and regulatory mandates (Malek & Gundaliya, 2021). This systematic alignment not only enhances the framework’s practical relevance but also fosters transparency, accountability and regulatory compliance within the PPP ecosystem.
Their feedback and insights provided valuable qualitative validation, enhancing the framework’s robustness and ensuring its suitability for assessing PPP project performance. This collaborative effort between experts with diverse backgrounds and expertise strengthened the framework’s credibility and effectiveness in evaluating PPP projects in Pakistan.
Despite the detailed methodology employed in this study, several limitations and opportunities for future research should be noted. First, though the RII is effective in ranking the factors, it may not fully capture the complexity of interdependent variables in PPP projects. Future studies could enhance understanding by integrating advanced statistical methods such as factor analysis or Garrett’s ranking technique to explore factor interactions and dependencies. The study utilised a relatively small sample size, based on participant availability and expertise in PPP projects, enhancing external validity but may limit the generalisability of results. Future research should consider expanding the sample size and diversifying participant demographics and project contexts to improve broader applicability. Though the IPAF predominantly focusses on PPP projects in Pakistan it can be applicable in other developing countries as well, its applicability may vary across different project types and geographical contexts. Validation through case studies across diverse sectors and regions is essential. Future research could also refine the framework by adapting it to specific project characteristics and local regulatory environments, incorporating feedback loops for iterative improvement based on real-world applications. Finally, critiques and advancements in methodology, as highlighted by recent literature (Johnson and LeBreton 2004; Vibhute et al. 2023), underscore the ongoing need for methodological refinements in utilising RII and other quantitative approaches in risk management studies. Future studies should explore these refinements to deepen the understanding of risk factors and management practices in PPP projects. In conclusion, while the IPAF represents a significant advancement in PPP project assessment frameworks, addressing these limitations and exploring future research avenues will enhance its applicability, effectiveness and contribution to infrastructure development practices on a broader scale.
The conclusions drawn from this study underscore the critical role of PPPs as a developmental tool, particularly in challenging economic conditions like those in Pakistan. It emphasises the necessity for governments to raise awareness among policymakers, decision-makers and the public to ensure the success of PPP transactions. Monitoring and evaluation (M&E) of PPP projects emerge as vital components in delivering VfM, yet the existing literature lacks IPAFs both locally and globally. This study addresses this gap by introducing a systematic approach to assess PPP project performance, focussing on risk measurement across various project phases. Through a rigorous literature review, 99 critical risk factors were identified and 43 were selected for further analysis. These risks encompass stakeholder, construction, operation, financial, legal, governmental and environmental dimensions, aligning with three KPIs: cost, time and quality, supported by 16 PMs, which were shortlisted from 114 PMs, validated through expert opinion and applicable across different project phases and contexts. Structured interviews with experienced PPP industry professionals validated the significance of identified risks and PMs, highlighting key concerns such as inflation, force majeure events and project completion delays. Integrating these findings, a performance assessment framework was developed, emphasising the importance of assessing project phases, PMs and associated risks.
The proposed framework provides a structured approach for evaluating PPP project outcomes and success across various dimensions such as financial, environmental and social factors. It emphasises the importance of continuous M&E throughout the project lifecycle to ensure optimal performance. By explicitly linking risks to PMs, the framework enables stakeholders to develop effective risk mitigation strategies. In conclusion, the study underscores the significance of PPP performance evaluation in improving project outcomes, mitigating risks and promoting transparency and accountability in infrastructure development initiatives. The proposed IPAF will serve as a valuable tool for stakeholders navigating PPP implementations, fostering more effective and sustainable development efforts in Pakistan and beyond.
The IPAF for PPP projects applies to various stakeholders, including government agencies, private investors and project managers. To effectively utilise the IPAF, stakeholders must have access to comprehensive project data, encompassing financial records, stakeholder feedback and expert opinions to validate risks and PMs. Additionally, conducting a comprehensive literature review ensures alignment with global best practices, while understanding government policies and regulations provides essential context. The framework’s effectiveness is further enhanced through meticulous project documentation, monitoring tools and active stakeholder engagement. By leveraging these resources, stakeholders can assess PPP project outcomes, pinpoint areas for improvement and make well-informed decisions to optimise project performance and achieve developmental objectives. It’s worth noting that IPAF operates on a time-bound approach, where project evaluation occurs within predefined time intervals, such as quarterly or annually, ensuring ongoing assessment and improvement.
However, the study acknowledges limitations in the framework, such as its focus on water supply, sewerage, road and bridge construction projects. Future research opportunities include expanding case studies, contextualising the framework across different project types and countries and accommodating stakeholder-specific amendments for broader applicability. The proposed performance evaluation framework should undergo validation through case studies and practical implementations to assess its effectiveness and applicability. Continuous monitoring and adaptation of the framework are essential to align with evolving project requirements and changing circumstances. Establishing a standardised rating scale based on this framework would enhance comparability and evaluation consistency across PPP projects. Conducting thorough pre-feasibility studies is imperative to ensure cost-efficient PPP projects. Revenue generation projections must be meticulously examined, considering potential force majeure events and their impact on revenue streams. Mitigation strategies should be developed early in the project lifecycle to minimise cost overruns and ensure VfM for project stakeholders. Rigorous monitoring of time-based performance should occur during project execution and construction stages. Factors such as force majeure events, third-party delays and governmental policies should be accounted for during project planning, with appropriate mitigation strategies proposed to mitigate delays. Concession periods should be optimised considering revenue generation factors and potential disruptions. Quality assurance measures should be implemented during project execution and operation stages, focussing on identifying and rectifying defects and problems promptly. Indirect factors impacting project quality, including transparency, effectiveness, efficiency, stakeholder communication and procurement processes, should also be carefully evaluated. Risks affecting these aspects should be systematically addressed to uphold project quality standards.