COMPARATIVE EVALUATION OF PROJECT COST FORECASTING METHODS BASED ON EARNED VALUE MANAGEMENT
|
TABLE OF CONTENTS INTRODUCTION 8
CHAPTER 1. EXISTING RESEARCH ON EARNED VALUE MANAGEMENT 10
1.1. Project control using earned value management 10
1.2. Forecasting techniques based on earned value management 21
1.3. Identification of research gaps in the contemporary research on earned value
management 29
1.4. Summary of Chapter 1 30
CHAPTER 2. RESEARCH FRAMEWORK FOR IDENTIFYING THE MOST
ACCURATE AND PRECISE COST FORECASTING TECHNIQUE 32
2.1. Research problem 32
2.2. Research design 34
2.3. Project data 40
2.4. Summary of Chapter 2 43
CHAPTER 3. RESULTS OF THE FORECASTING AND IDENTIFICATION OF
INFLUENCING FACTORS 44
3.1. Modified index-based method incorporating Gompertz growth function
forecasting results 44
3.2. XSM forecasting results 47
3.3. Comparative evaluation of the forecasted results 53
3.4. Managerial implications 56
3.5. Summary of Chapter 3 58
CONCLUSION 59
REFERENCES 62
Appendix
CHAPTER 1. EXISTING RESEARCH ON EARNED VALUE MANAGEMENT 10
1.1. Project control using earned value management 10
1.2. Forecasting techniques based on earned value management 21
1.3. Identification of research gaps in the contemporary research on earned value
management 29
1.4. Summary of Chapter 1 30
CHAPTER 2. RESEARCH FRAMEWORK FOR IDENTIFYING THE MOST
ACCURATE AND PRECISE COST FORECASTING TECHNIQUE 32
2.1. Research problem 32
2.2. Research design 34
2.3. Project data 40
2.4. Summary of Chapter 2 43
CHAPTER 3. RESULTS OF THE FORECASTING AND IDENTIFICATION OF
INFLUENCING FACTORS 44
3.1. Modified index-based method incorporating Gompertz growth function
forecasting results 44
3.2. XSM forecasting results 47
3.3. Comparative evaluation of the forecasted results 53
3.4. Managerial implications 56
3.5. Summary of Chapter 3 58
CONCLUSION 59
REFERENCES 62
Appendix
Forecasting is an indispensable part of project management both in the planning stage of a project when the necessary budget must be determined and throughout the project execution since the information provided by means of estimation on the needed funds serves as a basis for effective decision-making. Without forecasting that provides sound and reasonable estimates about project’s future performance the project might fail to meet its primary quality, schedule and cost objectives. Empirical research demonstrates that the more project progresses, the less of an impact corrective actions have on the project performance (Vanhoucke, 2014). Therefore, availability of a reliable forecasting instrument especially in the early stages of the project’s lifespan is crucial to the project success.
Forecasting as a part of an integrated project management control system aids project managers in making timely and appropriate decisions about cost risks that might impact their in-progress projects (Caron et al., 2016). Other project stakeholders, those being project owner and senior management of the company, can in turn objectively assess project performance when equipped with objective and consistent project performance indicators (Fleming and Koppelman, 2010). Moreover, reliable estimates on project progress contribute to spending optimization and smooth operations within an organization by detecting possible negative tendencies threatening project’s success (Homutinnikova, 2009).
Earned value management (EVM) is a well-recognized project management tool that integrates project cost, schedule and scope dimensions into a single framework. It is widely applied for measuring and analyzing project actual status against its baseline, and for providing estimates of project cost and duration at completion. (PMBOK, 2013). In particular, EVM is used to compute cost estimate at completion (EAC), a top-down estimate of the project total cost based on the project's current status.
Contemporary literature provides for a number of enhancements to conventional EVM forecasting approaches. These EVM-based methods can be classified as deterministic, stochastic and fuzzy (Willems and Vanhoucke, 2015). While the latter techniques address the vagueness of information inherent in projects, stochastic methods aim at integrating risk analysis into EVM framework. Deterministic approaches to forecasting build upon the established EVM system and, thus, share its most prominent advantages, namely, ability to adapt to project characteristics regardless of project size or industry, in which it is executed, and flexibility in use, e.g. the capability to be applied on any level of the work breakdown structure (Fleming and Koppelman, 2010).
EVM is used extensively for both project planning and monitoring and control (PMBOK, 2013). The current study focuses on the latter domain. Considering the area of application and the aforementioned advantages of deterministic approaches, two most promising contemporary deterministic techniques to project cost forecasting were chosen for comparative evaluation in this study. More specifically, the methods under investigation are exponential smoothing method (XSM) (Batselier and Vanhoucke, 2017) and modified index-based method incorporating Gompertz growth model (GGM) (Narbaev and De Marco, 2014). The comparative evaluation is carried out by assessing the methods’ accuracy and precision.
The research is conducted on the data of 31 construction and 18 software development projects. Construction and IT industry were chosen with the aim to test the methods’ applicability in divergent circumstances.
The goal of the research is to identify the most accurate and precise deterministic EVM- based approach to project final cost forecasting and the factors affecting its performance.
To reach the stated goal of the study the following research questions are to be answered: Q1. Which deterministic approach (XSM or GGM) produces the most accurate and precise forecasts?
Q2. Are early and middle stage levels of forecasting accuracy commensurate with the late stage level?
Q3. To what degree does the forecasting accuracy differ between construction and IT sectors and to which factors can that be attributable?
The thesis is organized in the following way. The first section of Chapter 1 summarizes up-to-date information on EVM framework, its fundamental principles and metrics, and implementation in industry. In addition, EVM limitations present in contemporary literature are discussed. The second section is devoted to various time and cost forecasting techniques based on EVM metrics. The chapter finishes with identification of research gaps in the state-of-the-art literature on EVM.
Chapter 2 outlines research problem and presents methodological design of the research. The forecasting techniques under investigation are further elaborated on with the focus on application procedures. Finally, the project data is provided.
Chapter 3 provides for results of the research and their analysis as well as managerial implications of the study.
The thesis finishes with conclusion including main findings, contribution of the study, its limitations and suggestions for future research.
Forecasting as a part of an integrated project management control system aids project managers in making timely and appropriate decisions about cost risks that might impact their in-progress projects (Caron et al., 2016). Other project stakeholders, those being project owner and senior management of the company, can in turn objectively assess project performance when equipped with objective and consistent project performance indicators (Fleming and Koppelman, 2010). Moreover, reliable estimates on project progress contribute to spending optimization and smooth operations within an organization by detecting possible negative tendencies threatening project’s success (Homutinnikova, 2009).
Earned value management (EVM) is a well-recognized project management tool that integrates project cost, schedule and scope dimensions into a single framework. It is widely applied for measuring and analyzing project actual status against its baseline, and for providing estimates of project cost and duration at completion. (PMBOK, 2013). In particular, EVM is used to compute cost estimate at completion (EAC), a top-down estimate of the project total cost based on the project's current status.
Contemporary literature provides for a number of enhancements to conventional EVM forecasting approaches. These EVM-based methods can be classified as deterministic, stochastic and fuzzy (Willems and Vanhoucke, 2015). While the latter techniques address the vagueness of information inherent in projects, stochastic methods aim at integrating risk analysis into EVM framework. Deterministic approaches to forecasting build upon the established EVM system and, thus, share its most prominent advantages, namely, ability to adapt to project characteristics regardless of project size or industry, in which it is executed, and flexibility in use, e.g. the capability to be applied on any level of the work breakdown structure (Fleming and Koppelman, 2010).
EVM is used extensively for both project planning and monitoring and control (PMBOK, 2013). The current study focuses on the latter domain. Considering the area of application and the aforementioned advantages of deterministic approaches, two most promising contemporary deterministic techniques to project cost forecasting were chosen for comparative evaluation in this study. More specifically, the methods under investigation are exponential smoothing method (XSM) (Batselier and Vanhoucke, 2017) and modified index-based method incorporating Gompertz growth model (GGM) (Narbaev and De Marco, 2014). The comparative evaluation is carried out by assessing the methods’ accuracy and precision.
The research is conducted on the data of 31 construction and 18 software development projects. Construction and IT industry were chosen with the aim to test the methods’ applicability in divergent circumstances.
The goal of the research is to identify the most accurate and precise deterministic EVM- based approach to project final cost forecasting and the factors affecting its performance.
To reach the stated goal of the study the following research questions are to be answered: Q1. Which deterministic approach (XSM or GGM) produces the most accurate and precise forecasts?
Q2. Are early and middle stage levels of forecasting accuracy commensurate with the late stage level?
Q3. To what degree does the forecasting accuracy differ between construction and IT sectors and to which factors can that be attributable?
The thesis is organized in the following way. The first section of Chapter 1 summarizes up-to-date information on EVM framework, its fundamental principles and metrics, and implementation in industry. In addition, EVM limitations present in contemporary literature are discussed. The second section is devoted to various time and cost forecasting techniques based on EVM metrics. The chapter finishes with identification of research gaps in the state-of-the-art literature on EVM.
Chapter 2 outlines research problem and presents methodological design of the research. The forecasting techniques under investigation are further elaborated on with the focus on application procedures. Finally, the project data is provided.
Chapter 3 provides for results of the research and their analysis as well as managerial implications of the study.
The thesis finishes with conclusion including main findings, contribution of the study, its limitations and suggestions for future research.
Forecasting plays a crucial role in the project management both in the planning stage of a project when the necessary budget must be determined and throughout the project execution since the information provided by means of estimation on the needed budget serves as a basis for effective decision-making. Without forecasting that provides sound and reasonable estimates about project’s future performance the project might fail to meet its primary quality, schedule and cost objectives.
With that rationale in mind, the present research was conducted with the aim of identifying the most accurate and precise deterministic EVM-based forecasting technique to project cost forecasting. The EVM approach to project cost control was chosen due to comparative advantages it offers respect to classic budget-costs evaluation approach and wide implementation in practice.
The contemporary literature on EVM cost control was investigated and two most promising deterministic EVM-based approaches to project cost forecasting were identified, those being XSM and GGM methods. The forecasting techniques were applied to 31 project from the construction industry and 18 projects from IT industry. The industries were chosen with the aim to test models’ applicability in different situations. Moreover, literature review demonstrated insufficiency of studies on software projects.
Evaluation approach consisted of assessment of models’ accuracy and precision. The accuracy was measured by means of MAPE technique, which is a mean absolute percentage error of forecasted results against real values. Precision was evaluated by SD, i.e. the dispersion of the values of prediction errors from the average forecast within the population. The lower values of both measures indicate better accuracy and precision of the model, respectively. The evaluation of accuracy was performed on two levels, namely, project level and stage level.
Comparative evaluation on the forecasting accuracy and precision of the both methods demonstrated the superiority of XSM respect to GGM. Moreover, XSM was also compared with the traditional EVM forecasting approach, namely EVM-CPI in terms of forecasting accuracy.
The results demonstrated that XSM is a preferable approach to cost forecasting in the construction industry respect to both GGM and the traditional EVM (EVM-CPI) approaches. XSM proved to be more accurate in both specifications (individual and overall) and in the additional setting, in which the most recent project in the sample was deemed ongoing. Moreover, XSM was relatively more accurate at different stages of project execution as well. Apart from that, the research demonstrated that construction projects on average have optimal b values in the range of 0,1 and 0,2, and that forecasting accuracy was commensurate in this region. Further on, the study revealed that first, smaller construction projects demonstrate worse budget adherence respect to bigger ones, and second, smaller projects have optimal b values different from zero and unity.
These findings imply that XSM is especially beneficial when applied to smaller projects. Finally, given low dispersion of forecasting errors in the sample of construction projects, optimal b values obtained over the sample of historical projects can be used when forecasting cost outcomes of in-progress projects in the construction industry.
With regards to software development projects, XSM outperformed GGM in both individual and overall specifications both on project and stage levels. XSM outperformed EVM- CPI in individual project specification both on project and stage levels but fell short on the overall level. This finding along high dispersion of forecasting errors in the sample of software development projects implies that incorporating the performance of historical projects - expressed as an optimal b obtained over the sample of projects conducted in the past - to forecast cost outcome of in-progress project is not advisable in IT industry. Instead, using only information on current performance yields better results. Therefore, for forecasting cost outcomes of in-progress projects it is recommended to deploy conventional EVM-CPI forecasting. Given XSM’s higher sensitivity to experience-driven learning that occurs along the project’s progress and the effect of corrective management actions respect to EVM-CPI it is recommended to deploy XSM for performance assessment of historical projects to derive factors affecting project performance and further associate them with optimal b values. In prospect, XSM can be applied following a qualitative approach.
The contribution of the study is fourfold. First of all, two state-of-the-art and conventional approaches to project cost forecasting were evaluated in comparison respect to forecasting accuracy and precision. Moreover, this study added up by verifying the aforementioned methods on larger number of construction projects and software development projects compared to the previous studies: 1 IT project and 20 construction projects in (Batselier and Vanhoucke, 2017), 9 construction projects in (Narbaev and De Marco, 2014b). Thirdly, in this research XSM was tested in the additional setting. And finally, the decision support tool in the form of MS Excel spreadsheet based on XSM (Appendix 3) was developed with step-by-step application instruction.
From the practical point of view, XSM can be confidently applied in the construction industry considering the aforementioned research findings. Moreover, this method is especially beneficial for small-scale projects. In addition, the developed in this research XSM-based DSS can be used in conjunction with MS Project. With regards to IT industry careful cost-benefit analysis should be conducted when considering the use of XSM as its benefits are clear only in case it is applied to historical projects. For ongoing projects, it is advisable to deploy EVM-CPI.
The main limitation of the study is that all projects investigated in this research were conducted by Belgian companies. Application of the method in the economies with significantly different situations, e.g. in the developing countries like Russian Federation, will probably provide for different results. Thus, testing the method’s accuracy on projects conducted in different economic environment is the avenue for future research. Moreover, XSM applicability in IT industry can also be further investigated.
To cap it all, the study achieved its goal and provided answers to the stated research questions.
With that rationale in mind, the present research was conducted with the aim of identifying the most accurate and precise deterministic EVM-based forecasting technique to project cost forecasting. The EVM approach to project cost control was chosen due to comparative advantages it offers respect to classic budget-costs evaluation approach and wide implementation in practice.
The contemporary literature on EVM cost control was investigated and two most promising deterministic EVM-based approaches to project cost forecasting were identified, those being XSM and GGM methods. The forecasting techniques were applied to 31 project from the construction industry and 18 projects from IT industry. The industries were chosen with the aim to test models’ applicability in different situations. Moreover, literature review demonstrated insufficiency of studies on software projects.
Evaluation approach consisted of assessment of models’ accuracy and precision. The accuracy was measured by means of MAPE technique, which is a mean absolute percentage error of forecasted results against real values. Precision was evaluated by SD, i.e. the dispersion of the values of prediction errors from the average forecast within the population. The lower values of both measures indicate better accuracy and precision of the model, respectively. The evaluation of accuracy was performed on two levels, namely, project level and stage level.
Comparative evaluation on the forecasting accuracy and precision of the both methods demonstrated the superiority of XSM respect to GGM. Moreover, XSM was also compared with the traditional EVM forecasting approach, namely EVM-CPI in terms of forecasting accuracy.
The results demonstrated that XSM is a preferable approach to cost forecasting in the construction industry respect to both GGM and the traditional EVM (EVM-CPI) approaches. XSM proved to be more accurate in both specifications (individual and overall) and in the additional setting, in which the most recent project in the sample was deemed ongoing. Moreover, XSM was relatively more accurate at different stages of project execution as well. Apart from that, the research demonstrated that construction projects on average have optimal b values in the range of 0,1 and 0,2, and that forecasting accuracy was commensurate in this region. Further on, the study revealed that first, smaller construction projects demonstrate worse budget adherence respect to bigger ones, and second, smaller projects have optimal b values different from zero and unity.
These findings imply that XSM is especially beneficial when applied to smaller projects. Finally, given low dispersion of forecasting errors in the sample of construction projects, optimal b values obtained over the sample of historical projects can be used when forecasting cost outcomes of in-progress projects in the construction industry.
With regards to software development projects, XSM outperformed GGM in both individual and overall specifications both on project and stage levels. XSM outperformed EVM- CPI in individual project specification both on project and stage levels but fell short on the overall level. This finding along high dispersion of forecasting errors in the sample of software development projects implies that incorporating the performance of historical projects - expressed as an optimal b obtained over the sample of projects conducted in the past - to forecast cost outcome of in-progress project is not advisable in IT industry. Instead, using only information on current performance yields better results. Therefore, for forecasting cost outcomes of in-progress projects it is recommended to deploy conventional EVM-CPI forecasting. Given XSM’s higher sensitivity to experience-driven learning that occurs along the project’s progress and the effect of corrective management actions respect to EVM-CPI it is recommended to deploy XSM for performance assessment of historical projects to derive factors affecting project performance and further associate them with optimal b values. In prospect, XSM can be applied following a qualitative approach.
The contribution of the study is fourfold. First of all, two state-of-the-art and conventional approaches to project cost forecasting were evaluated in comparison respect to forecasting accuracy and precision. Moreover, this study added up by verifying the aforementioned methods on larger number of construction projects and software development projects compared to the previous studies: 1 IT project and 20 construction projects in (Batselier and Vanhoucke, 2017), 9 construction projects in (Narbaev and De Marco, 2014b). Thirdly, in this research XSM was tested in the additional setting. And finally, the decision support tool in the form of MS Excel spreadsheet based on XSM (Appendix 3) was developed with step-by-step application instruction.
From the practical point of view, XSM can be confidently applied in the construction industry considering the aforementioned research findings. Moreover, this method is especially beneficial for small-scale projects. In addition, the developed in this research XSM-based DSS can be used in conjunction with MS Project. With regards to IT industry careful cost-benefit analysis should be conducted when considering the use of XSM as its benefits are clear only in case it is applied to historical projects. For ongoing projects, it is advisable to deploy EVM-CPI.
The main limitation of the study is that all projects investigated in this research were conducted by Belgian companies. Application of the method in the economies with significantly different situations, e.g. in the developing countries like Russian Federation, will probably provide for different results. Thus, testing the method’s accuracy on projects conducted in different economic environment is the avenue for future research. Moreover, XSM applicability in IT industry can also be further investigated.
To cap it all, the study achieved its goal and provided answers to the stated research questions.