From 7bdda64445765733baed63bd97f007686c91b780 Mon Sep 17 00:00:00 2001 From: STEFANO GRASSI Date: Wed, 11 Sep 2024 17:38:53 +0000 Subject: [PATCH] Replace dsm060-cw2.ipynb --- dsm060-cw2.ipynb | 171 +++++++++++++++++++++++------------------------ 1 file changed, 84 insertions(+), 87 deletions(-) diff --git a/dsm060-cw2.ipynb b/dsm060-cw2.ipynb index 81e29ca..55c812e 100644 --- a/dsm060-cw2.ipynb +++ b/dsm060-cw2.ipynb @@ -5,86 +5,83 @@ "id": "0747037e-3ae6-466f-84b1-e2ef51c73e9a", "metadata": {}, "source": [ - "# Enhancing Probabilistic Time Series Forecasting with Conformal Prediction: An Empirical Study on Uncertainty Quantification\n", + "# Enhancing Uncertainity Quantification in Time Series Forecasting with Conformal Prediction: An Empirical Study\n", "\n", "## Motivation\n", "\n", - "In recent times of global turmoil, time series forecasting has become an increasingly important and relevant area of study. While forecasting point estimates remain fundamental, advances in computing, alongside significant events such as COVID-19, the Russia-Ukraine war, inflation and the Israeli crisis, have intensified the focus on probabilistic forecasting (Makridakis et al., 2018; Makridakis et al., 2020; Makridakis et al., 2022a). For example, in financial risk management, precise uncertainty estimates are crucial for effective portfolio management and for quantifying risks. Inaccurate prediction intervals can lead to misallocation of resources and increased risk exposure. Similarly, in healthcare and disaster management, accurate uncertainty quantification can improve resource allocation and crisis management (Makridakis & Bakas, 2016; Raftery, 2016).\n", + "In recent times of global turmoil, time series forecasting has become an increasingly important and relevant area of study. While forecasting point estimates remain fundamental, advances in computing, alongside unexpected events such as COVID-19, the Russia-Ukraine war, inflation and the Israeli crisis, have intensified the focus on probabilistic forecasting (Makridakis et al., 2018; Makridakis et al., 2020; Makridakis et al., 2022a). For example, in financial risk management, precise uncertainty estimates are fundamental for effective portfolio management and for quantifying risks. Inaccurate prediction intervals can lead to misallocation of resources and increased risk exposure. Similarly, in healthcare and disaster management, accurate uncertainty quantification can improve resource allocation and crisis management (Makridakis & Bakas, 2016; Raftery, 2016).\n", "\n", "Classical forecasting methods (Box & Jenkins, 1976) often face challenges in accurately estimating prediction intervals (Makridakis et al., 2018; Grushka-Cockayne & Jose, 2020) and typically rely on restrictive distributional assumptions. With the rise of Machine Learning (ML) in forecasting (Makridakis et al., 2022), which frequently involves black-box models, quantifying uncertainty has become increasingly complex, calling for a need to develop methods that address model misspecification while maintaining computational efficiency, especially for large and intricate ones (Martin et al., 2024).\n", "\n", - "Recent advancements in Machine Learning, particularly Conformal Prediction (CP), offer a distribution-free approach with guaranteed coverage to quantifying uncertainty. Introduced by Gammerman, Vovk and Vapnik (1998) and further extended to the time series domain, CP demonstrates promising capabilities for addressing the challenges faced in Probabilistic Forecasting. \n", + "Conformal Prediction (CP) offers a distribution-free approach with guaranteed coverage for quantifying uncertainty. Introduced by Gammerman, Vovk and Vapnik (1998), CP does not rely on strict distributional conditions, making it suitable for a wide range of models, especially Machine Learning ones. Extended to the time series domain, CP shows promise by providing adaptive prediction intervals that adjust with new data, addressing challenges like model misspecification and uncertainty, while ensuring reliable coverage.\n", "\n", - "Therefore, enhancing Probabilistic Time Series Forecasting methods with Conformal Prediction offers a novel opportunity to achieve more reliable prediction intervals, which is crucial in high-risk scenarios and in an era where Machine Learning is rapidly influencing the world.\n", + "Therefore, enhancing probabilistic time series forecasting methods with Conformal Prediction adds an extra layer of uncertainty quantification on top of existing models, offering a novel opportunity to achieve more accurate prediction intervals. This is particularly important in high-risk scenarios and in an era where Machine Learning is rapidly shaping decision-making processes.\n", "\n", "## Aim and Objectives\n", "\n", "### Aim\n", "\n", - "The aim of this project is to provide an in-depth literature review on Conformal Prediction (CP) in the context of uncertainty quantification in time series, and to offer additional empirical evidence on its effectiveness. This will be achieved by applying some of the latest CP methods to multiple datasets and benchmark models that attempt to replicate the conditions of the latest M5 uncertainty competition (Makridakis et al., 2022).\n", + "The aim of this project is to provide an in-depth literature review on Conformal Prediction (CP) in the context of uncertainty quantification in time series, and to offer additional empirical evidence on its effectiveness. This will be achieved by applying two of the latest CP methods on 2-3 models in time series forecasting to multiple datasets and benchmark models that attempt to replicate the conditions of the latest M5 uncertainty competition (Makridakis et al., 2022).\n", "\n", "### Objectives\n", "\n", "The objectives of the project are the following:\n", "\n", - "* **Review and analyze Conformal Prediction methods in the context of time series**: conduct a thorough literature review of current CP techniques in the context of time series.\n", - "* **Empirical test**: implement `ACI` and `EnbPI` on 2-3 time series forecasting models using 2-3 diverse datasets from the Monash Time Series Forecasting Archive that includes datasets and models used in the M5 uncertainty competition.\n", - "* **Evaluate and compare performance**: compare the prediction intervals generated by the CP-enhanced models with those from 2-3 benchmark methods. Use relevant evaluation metrics such as those used in to assess the accuracy and reliability of these prediction intervals.\n", - "* **Document and present findings**: summarise the results, highlighting the empirical performance of CP methods compared to benchmarks. Provide insights into the practical benefits and limitations of CP for uncertainty quantification in time series forecasting.\n", - "\n", + "* **Review and analyze Conformal Prediction methods in the context of time series**: conduct a thorough literature review of current CP techniques in time series.\n", + "* **Empirical test**: implement `ACI` and `EnbPI` on two to three base time series forecasting models using two to three diverse datasets from the Monash Time Series Forecasting Archive that includes datasets and models used in the M5 uncertainty competition.\n", + "* **Evaluate and compare performance**: compare the prediction intervals generated by the CP-enhanced models with those from six benchmark methods. Use relevant evaluation metrics to assess the accuracy and reliability of these prediction intervals.\n", + "* **Document and present findings**: summarise the results, highlighting the empirical performance of the base models enhanced by the two CP methods compared to benchmarks. Provide insights into the practical benefits and limitations of CP for uncertainty quantification in time series forecasting.\n", "\n", "## Stakeholders\n", "\n", - "The proposed project aims to benefit a wide range of stakeholders in both academic and practical domains.\n", + "The proposed project is designed to benefit a wide range of stakeholders across both academic and industry settings.\n", "\n", - "In academia, researchers might benefit from a thorough literature review alongside the additional empirical evidence provided by the project. This could contribute to the fields of Machine Learning and Forecasting by strengthening previous findings and encouraging further research on Conformal Prediction, specifically in the time series domain.\n", + "In academia, educators and students may find value in the comprehensive literature review and the empirical results generated by the project. This work can contribute to a clearer understanding of Conformal Prediction (CP) in time series forecasting and provide teaching material or evidence for the broader academic discourse. Although CP is an emerging field, this study could help contextualize its practical applications without claiming groundbreaking discoveries.\n", "\n", - "In industry, stakeholders across various sectors might gain from additional empirical tests on using Conformal Prediction to estimate more reliable prediction intervals. The existing literature on Conformal Prediction in the context of time series is still nascent and more empirical tests are needed to faciliate further adoption.\n", + "In industry, practitioners and analysts across various sectors may benefit from the additional empirical tests that demonstrate the use of CP to produce more reliable prediction intervals in time series forecasting. While CP is still relatively new in this domain, the project can serve as an example of how these methods can enhance decision-making, facilitating better adoption of CP for uncertainty quantification at the firm level.\n", "\n", - "Lastly, academic advisors and data providers are crucial stakeholders for the success of this project. Advisors will offer guidance on the project development and Data providers will facilitate access to the datasets necessary for testing and validating the proposed methods.\n", + "Finally, academic advisors and data providers play a critical role in the project’s success. Advisors will offer essential guidance throughout the development process, while data providers ensure access to the datasets required for thorough testing and validation of the methods used.\n", "\n", - "In summary, the project’s stakeholders include businesses, academic researchers, data providers and academic advisors.\n", + "In summary, the project’s stakeholders include educators, industry professionals, academic advisors and data providers.\n", "\n", "## Related Work\n", "\n", - "Conformal Prediction (CP) or Conformal Inference (CI) is a framework that leverages past data to predict confidence levels for new instances (Shafer et al., 2008). This framework addresses several challenges associated with traditional statistical methods and Machine Learning techniques such as sensitivity to assumptions, trade-offs between coverage and interval width and computational complexity (Manokhin, 2023). Despite its advantages, CP's reliance on the assumption of exchangeability poses challenges in time series applications, where observations are often dependent and distribution shifts occur.\n", + "Conformal Prediction (CP) is a framework that leverages past data to predict confidence levels for new instances (Shafer et al., 2008). This framework adds an extra layer of uncertainty quantification to traditional statistical methods and Machine Learning techniques, addressing challenges such as sensitivity to assumptions, trade-offs between coverage and interval width and computational complexity (Manokhin, 2023). However, CP’s reliance on exchangeability assumptions can be challenging in time series applications, where observations are dependent and distribution shifts occur.\n", "\n", - "Initial efforts to extend the CP framework to time series data focused on minimizing dependency assumptions that limit the exchangeability requirement. Balasubramanian et al. (2014) proposed addressing this by assuming that each observation depends only on those within a given lag $T \\in \\mathbb{N}$ or window, rather than all past observations. This modified CP framework has been adopted in several studies. For instance, Kath and Ziel (2021) applied it to short-term electricity price forecasting, finding it more reliable than methods such as quantile regression averaging and empirical error distribution approaches. Similarly, Wisniewski, Lindsay, and Lindsay (2020) used it for financial time series, showing that CP yields comparable or slightly better results than quantile regression in forecasting net positions. Additionally, Kowalczewski (2019) enhanced this approach by incorporating normalization in the context of energy forecasting.\n", + "Initial efforts to extend the CP framework to time series data focused on minimizing dependency assumptions that limit the exchangeability requirement. Balasubramanian et al. (2014) proposed addressing this by assuming that each observation depends only on those within a given lag $T \\in \\mathbb{N}$ or window, rather than all past observations. This modified CP framework has been adopted in several studies. For instance, Kath and Ziel (2021) applied it to short-term electricity price forecasting, finding it more reliable than methods such as quantile regression averaging and empirical error distribution approaches. Similarly, Wisniewski, Lindsay, and Lindsay (2020) used it for financial time series, showing that CP yields comparable or slightly better results than quantile regression in forecasting net positions.\n", "\n", "More recent work has broadened CP to address a variety of distribution shifts and dependency structures, including those found in time series and other complex data settings trying to move beyond exchangability (Tibshirani et al., 2019; Podkopaev & Ramdas, 2021; Barber et al., 2022). \n", "\n", "In particular, several different approaches have emerged that don't rely on exchangability. \n", "\n", - "Xu and Xie (2021) developed Ensemble batch Prediction Intervals (`EnbPI`), a method that updates residuals of ensemble predictors during prediction to more accurately calibrate prediction intervals without splitting data allowing `EnbPI` to maintain desired $1 - \\alpha$ coverage for different types of time series. `EnbPI` rapidly increased popularity and is currently implemented in the open-source libraries Amazon Fortuna (Detommaso et al., 2023) and MAPIE (Taquet et al., 2022). Moreover, Building on top of Barber et al. (2022), due to limitations of `EnbPI`, Xu and Xie (2023) introduced Sequential Predictive Conformal Inference ( `SPCI`) which utilizes the feedback structure of prediction residuals in the sequential prediction problem to obtain desired coverage.\n", + "Xu and Xie (2021) developed Ensemble batch Prediction Intervals (`EnbPI`), a method that updates residuals of ensemble predictors during prediction to more accurately calibrate prediction intervals without splitting data allowing `EnbPI` to maintain desired $1 - \\alpha$ coverage for different types of time series. `EnbPI` and its second version `EnbPI V2` rapidly increased popularity and is currently implemented in the open-source libraries Amazon Fortuna (Detommaso et al., 2023) and MAPIE (Taquet et al., 2022). Moreover, Building on top of Barber et al. (2022), due to limitations of `EnbPI`, Xu and Xie (2023) introduced Sequential Predictive Conformal Inference ( `SPCI`) which utilizes the feedback structure of prediction residuals in the sequential prediction problem to obtain desired coverage.\n", "\n", "On the other hand, Adaptive Conformal Inference (`ACI`), proposed by Gibbs and Candès (2021) and its subsequent variants (Gibbs and Candès, 2022; Zaffran et al., 2022; Feldman et al., 2023; Angelopoulos et al., 2023) offer a significant advancement by providing approximately calibrated prediction intervals by adjusting a time-varying nominal miscoverage rate or employs an online gradient descent algorithm for calibration. Additionally, Bellman Conformal Inference (`BCI`), introduced by Yang et al. (2024), addresses the issue of `ACI` of not explicitly optimize average interval lengths by adjusting parameters to balance interval length and coverage rates, potentially offering more efficient intervals compared to `ACI` when nominal intervals are poorly calibrated.\n", "\n", "### Gaps and Shortcomings\n", "\n", - "A notable shortcoming in existing Conformal Prediction (CP) methods is the limited empirical validation across diverse scenarios. While theoretical advancements are significant, there is a need for practical evaluations to confirm effectiveness. However, these advancements lack comprehensive empirical testing across different forecasting models and datasets in real-world situations, which is essential for broader adoption. Additionally, there is a gap in the literature regarding a thorough review of the latest developments in CP, particularly in the context of time series forecasting. Although the proposed models have their own limitations, which researchers are actively addressing, this project focuses on addressing the missing aspects—conducting an in-depth literature review and empirical demonstration.\n", + "A notable shortcoming in existing Conformal Prediction (CP) methods is the limited empirical validation across diverse scenarios. While theoretical advancements are significant, there is a need for practical evaluations to confirm effectiveness. However, these advancements lack comprehensive empirical testing across different forecasting models and datasets in real-world situations, which is essential for broader adoption. Additionally, there is a gap in the literature regarding a thorough review of the latest developments in CP in time series forecasting. Although the proposed models have their own limitations, which researchers are actively addressing, this project focuses on addressing the missing aspects—conducting an in-depth literature review and empirical demonstration.\n", "\n", "## Data\n", "\n", - "To achieve the project's aim of enhancing uncertainty quantification with Conformal Prediction (CP) in time series forecasting, datasets will be sourced from the Monash Time Series Forecasting Archive (Godahewa et al., 2021). Curated by leading expert Rob J. Hyndman and the Monash University team, this archive offers a meticulously curated collection of time series datasets essential for benchmarking and evaluating forecasting models. Monash University’s involvement in the M-competitions further underscores the archive's authority and relevance.\n", + "To achieve the project's aim of enhancing uncertainty quantification with CP in time series Forecasting, datasets will be sourced from the Monash Time Series Forecasting Archive (Godahewa et al., 2021). Curated by leading expert Rob J. Hyndman and the Monash University team, this archive offers a meticulously curated collection of time series datasets essential for benchmarking and evaluating forecasting models. Monash University’s involvement in the M-competitions further underscores the archive's authority and relevance for this project.\n", "\n", "### The Monash Time Series Forecasting Archive \n", "\n", - "The Monash Time Series Forecasting Archive comprises 30 distinct datasets, with 58 variations based on different time frequencies and the handling of missing values. This diversity will provide a comprehensive testing ground across various real-world scenarios, including finance, energy and healthcare. The archive’s meticulous curation will ensure high-quality data, and its GitHub repository will offer detailed instructions for integrating and evaluating new forecasting models against baseline metrics. This approach will guarantee data quality and model comparability, making the archive an ideal resource for the project’s empirical analysis.\n", + "The Monash Time Series Forecasting Archive comprises 30 distinct datasets, with 58 variations based on different time frequencies and the handling of missing values. This diversity will provide a comprehensive testing ground across various real-world scenarios, including banking, energy and healthcare. The archive’s meticulous curation will ensure high-quality data, and its GitHub repository will offer detailed instructions for integrating and evaluating new forecasting models against baseline metrics. This approach will guarantee data quality and model comparability, making the archive an ideal resource for the project’s empirical analysis.\n", "\n", "## Methods\n", "\n", - "To achieve the aims of this project, a quantitative study with two Conformal Prediction (CP) methods will be employed. The selected methods are:\n", + "To achieve the project aims, a quantitative study will be conducted using two Conformal Prediction (CP) methods alongside a literature review. The chosen methods are:\n", "\n", - "* Ensemble Bootstrap Prediction Interval (`EnbPI`): Developed by Xu & Xie (2021), EnbPI addresses the challenge of non-exchangeable data in time series by adding a sequential aspect to the CP framework. This method is designed for robust uncertainty quantification in time series forecasting and is accessible through the open-source MAPIE library, making it an ideal candidate for empirical testing. EnbPI's design ensures that prediction intervals remain valid even in the presence of dependent data.\n", - "* Adaptive Conformal Inference (`ACI`): Proposed by Gibbs and Candès (2021), ACI dynamically adjusts prediction intervals to accommodate distribution shifts within time series data. Its ability to adapt to shifts in the data distribution without sacrificing the validity of prediction intervals makes it particularly suited for time series with general dependency structures. The method’s flexibility is further enhanced by its extension, AgACI, which eliminates the need for tuning hyperparameters by using online expert aggregation. ACI has been shown to improve the efficiency of prediction intervals in time series settings, as demonstrated in synthetic experiments and real-world applications like electricity price forecasting.\n", - " \n", - "These methods were previously compared in a study by Zaffran et al. (2022), which serves as an inspiration for this project. The study highlighted the strengths and limitations of each method, underscoring their potential for further empirical investigation in the context of time series forecasting. Moreover, due to their current traction, it makes sense to use these methods rather than others proposed in the literature.\n", + "* **Ensemble Bootstrap Prediction Interval (`EnbPI`)**: Developed by Xu & Xie (2021), `EnbPI` is designed for non-exchangeable time series data. It works by generating multiple bootstrap samples from the training data and fitting the regression model to each sample. The prediction intervals are then constructed by aggregating the outputs from these models. This approach allows `EnbPI` to adapt to changes in seasonality and trends without needing to refit the model, ensuring robust uncertainty quantification for dependent data.\n", + "* **Adaptive Conformal Inference (`ACI`)**: Proposed by Gibbs and Candès (2021), `ACI` addresses temporal distribution shifts by adapting prediction intervals based on a dynamically updated miscoverage level. It employs an online procedure with recursive adjustment of coverage levels to handle varying data distributions, maintaining accurate coverage even when the underlying data generating process changes.\n", "\n", - "In addition to these CP-based methods, traditional methods commonly used in probabilistic forecasting will be employed for comparison. These conventional approaches will provide a baseline against which the performance of CP methods can be measured, ensuring a comprehensive evaluation of their effectiveness in uncertainty quantification.\n", + "These CP methods were selected due to their demonstrated effectiveness, relevance for time series forecasting and accessibility via the MAPIE library using Python, which facilitates rigorous reproduction and testing across various scenarios. Their suitability is further validated by the comparative study conducted by Zaffran et al. (2022), which evaluated their strengths and limitations. This study highlights the potential of `EnbPI` and `ACI` for advancing empirical research in time series forecasting.\n", "\n", - "The availability of code for all two CP methods ensures they can be rigorously tested against these traditional benchmark models. Each method represents a different approach to uncertainty quantification in time series forecasting, offering valuable insights into their applicability and effectiveness across different scenarios compared to existing methods.\n", + "Base models for the study will be chosen through a comprehensive literature review. After selecting and optimizing 2-3 forecasting models, the CP methods (`EnbPI` and `ACI`) will be applied to produce prediction intervals. Their performance will be compared against benchmark models and evaluation metrics from the M5 uncertainty competition to ensure a thorough assessment of uncertainty quantification methods.\n", "\n", - "## Work plan\n", + "## Work Plan\n", "\n", "The proposed project will be completed over two months and is divided into two main phases: Phase 1 and Phase 2, with the work in Phase 1 informing Phase 2.\n", "\n", @@ -92,112 +89,113 @@ "\n", "#### Introduction Writing\n", "\n", - "* Key Step: Drafting the introduction to outline objectives, background and significance.\n", - "* Duration: this step is estimated to take few days, depending on how detailed the introduction needs to be. Further revisions are required to include a quick summary of key findings. \n", + "* **Key step**: drafting the introduction to outline objectives, background and significance.\n", + "* **Duration**: this step is estimated to take few days, depending on how detailed the introduction needs to be. Further revisions are required to include a quick summary of key findings. \n", "\n", - "#### Literature review\n", + "#### Literature Review\n", "\n", - "* Key step: Conducting a comprehensive review of existing research to understand the current state of knowledge.\n", - "* Duration: this step is estimated to take 1-2 weeks, depending on the volume of literature and depth of review required.\n", + "* **Key step**: conducting a comprehensive review of existing research to understand the current state of knowledge.\n", + "* **Duration**: this step is estimated to take 1-2 weeks, depending on the volume of literature and depth of review needed.\n", "\n", "#### Data Collection and Assessment\n", "\n", - "* Key Step: Gathering relevant data from the Monash Archive and assessing its suitability for the experimental setup.\n", - "* Duration: given that the Monash Time Series Archive does include the few days, depending on the availability and accessibility of the data.\n", + "* **Key step**: gathering relevant datasets from the Monash Archive and assessing their suitability for the experimental setup.\n", + "* **Duration**: it takes few hours.\n", "\n", "#### Methodology Development\n", "\n", - "* Key Step: Developing the research methodology, including techniques and processes.\n", - "* Duration: 1 week, as it involves planning and detailing the approach for the experimental phase.\n", + "* **Key step**: developing the research methodology, including techniques and processes.\n", + "* **Duration**: 1 week, as it involves planning and detailing the approach for the experimental phase.\n", "\n", "#### Validation\n", "\n", - "* Key Step: Validating research questions, methodology, and data assessment to ensure readiness for Phase 2.\n", - "* Duration: few days, to ensure that the methodology and data are robust and feasible.\n", + "* **Key step**: validating research questions, methodology and data assessment to ensure readiness for Phase 2.\n", + "* **Duration**: few days, to ensure that the methodology and data are robust and feasible.\n", "\n", "### Phase 2 - Month 2\n", "\n", "#### Data Processing\n", "\n", - "* Key Step: Cleaning and preparing data for analysis.\n", - "* Duration: few hours.\n", + "* **Key step**: cleaning and preparing data for analysis. Although the data are already curated, this phase may involve additional preprocessing and transformation, which could take longer based on the specific tasks.\n", + "* **Duration**: Several hours to a few days, depending on the complexity of the preprocessing requirements.\n", + "Model Building and Testing.\n", "\n", - "#### Model Building and Test\n", + "#### Experimental Setup\n", "\n", - "* Key Step: Developing and testing models based on the validated methodology.\n", - "* Duration: 1-2 weeks, as it involves iterative testing and refinement of models.\n", + "* **Key step**: Choosing base models, optimizing them, applying CP methods and comparing the results with benchmarks across several datasets. This phase involves iterative testing, model refinement and comprehensive evaluation, which might be time-consuming.\n", + "* **Duration**: 1 week to 1.5 weeks, allowing for sufficient time to complete all the steps, the detailed comparisons and adjustments.\n", "\n", "#### Analysis\n", "\n", - "* Key Step: Analyzing results and developing conclusions.\n", - "* Duration: 1 week, as it involves interpreting results and synthesizing findings.\n", + "* **Key step**: analyzing results and developing conclusions.\n", + "* **Duration**: 1 week, as it involves interpreting results and synthesizing findings.\n", "\n", "#### Conclusion Writing\n", "\n", - "* Key Step: Drafting the conclusion to summarize findings, discuss implications, and suggest future research.\n", - "* Duration: A few days, as it summarizes the project’s outcomes and implications.\n", + "* **Key step**: drafting the conclusion to summarize findings, discuss implications and suggest future research.\n", + "* **Duration**: a few days, as it summarizes the project’s outcomes and implications.\n", "\n", "#### Key Milestones for Evaluation\n", "\n", - "**End of Week 1 (Phase 1):**\n", - "* **Milestone:** Draft the Introduction.\n", - " * Task: Draft of the introduction and begin the literature review.\n", + "**End of Week 1 (Phase 1)**:\n", + "* **Milestone**: draft the introduction.\n", + " * Task: draft of the introduction and begin the literature review.\n", "\n", - "**End of Week 2 (Phase 1):**\n", - "* **Milestone:** Completion of Literature review.\n", - " * Task: Identify research gaps and complete the literature review.\n", + "**End of Week 2 (Phase 1)**:\n", + "* **Milestone**: completion of Literature review.\n", + " * Task: identify research gaps and complete the literature review.\n", "\n", - "**End of Week 3 (Phase 1):**\n", - "* **Milestone:** Completion of Data Collection and Methodology Development.\n", - " * Task: Collect and assess data, and develop the methodology.\n", + "**End of Week 3 (Phase 1)**:\n", + "* **Milestone:** completion of data collection and methodology development.\n", + " * Task: collect and assess data and develop the methodology.\n", "\n", - "**End of Week 4 (Phase 1):**\n", - "* **Milestone:** Validation of Research Framework.\n", - " * Task: Validate the research questions, methodology, and data assessment.\n", + "**End of Week 4 (Phase 1)**:\n", + "* **Milestone:** validation of research framework.\n", + " * Task: validate the research questions, methodology and data assessment.\n", "\n", - "**End of Week 5 (Phase 2):**\n", - "* **Milestone:** Completion of Data Processing.\n", - " * Task: Clean the data and prepare it for analysis.\n", + "**End of Week 5 (Phase 2)**:\n", + "* **Milestone:** completion of data processing.\n", + " * Task: clean the data and prepare it for analysis.\n", "\n", - "**End of Week 6 (Phase 2):**\n", - "* **Milestone:** Completion of model building and testing.\n", - " * Task: Build and test initial models.\n", + "**End of Week 6 (Phase 2)**:\n", + "* **Milestone:** completion of model building and testing.\n", + " * Task: build and test initial models.\n", "\n", "**End of Week 7 (Phase 2):**\n", - "* **Milestone:** Completion of Analysis.\n", - " * Task: Analyze results and develop initial findings.\n", + "* **Milestone:** completion of analysis.\n", + " * Task: analyze results and develop initial findings.\n", "\n", "**End of Week 8 (Phase 2):**\n", - "* **Milestone:** Completion of Conclusion Writing and Project Wrap-Up.\n", - " * Task: Draft the conclusion, complete the final editing, revisions, proofreading, and finalize the project report.\n", + "* **Milestone:** completion of conclusion and project wrap-pp.\n", + " * Task: draft the conclusion, complete the final editing, revisions, proofreading and finalize the project report.\n", "\n", "## Risk assessment\n", "\n", - "Effective risk management is crucial for the success of the proposed project. Three project-based risks have been identified.\n", + "Effective risk management is paramount for the success of the proposed project. Three project-based risks have been identified.\n", "\n", - "### Availability of Resources\n", + "### Data Quality and Integrity\n", "\n", - "* Risk: Types of might be unavailable or incomplete, and necessary hardware might not meet technical requirements.\n", - "* Mitigation: Validation of data acquisition sources was conducted to ensure reliability, thus the choice of Monash Time Series Archive. Hardware requirements have been pre-assessed ensuring they can run on the hardware currently in use which is Mac M1 (2020).\n", + "* **Risk**: there may be issues with data quality or integrity, such as missing or inconsistent data.\n", + "* **Mitigation**: ensure that data is sourced from the Monash Time Series Archive, known for its high-quality and well-curated datasets. Implement thorough data cleaning and preprocessing steps to address any issues discovered during the initial review.\n", "\n", - "### Scope Creep\n", + "### Resource Constraints\n", "\n", - "* Risk: The scope of the project could lead to an unmanageable workload such as extensive data preprocessing.\n", - "* Mitigation: The project’s aims and objectives will be adjusted to a nicher case if needed without compromising the core research question. This approach maintains focus while ensuring the project remains feasible.\n", + "* **Risk**: running complex time series models and processing large datasets may be resource-intensive, potentially leading to performance or time constraints.\n", + "* **Mitigation**: prioritize simpler and faster models to ensure computational efficiency. Continuously monitor resource usage and make necessary adjustments to stay within the project's timeframe and computational limits while training and testing models locally.\n", "\n", - "### Feasibility and Complexity\n", + "### Complexity and Feasibility\n", "\n", - "* Risk: The complexity of the selected models may exceed current skills, leading to delays.\n", - "* Mitigation: Iterative project planning will allow the focus to be narrowed if necessary. For example, focusing on a specific industry could reduce the number of datasets and simplify the models.\n", + "* **Risk**: integrating two Conformal Prediction methods with multiple base models and benchmarking across various datasets could pose challenges in terms of feasibility and execution.\n", + "* **Mitigation**: manage the project iteratively to address emerging challenges. Adjust the scope if needed to simplify models or datasets, ensuring alignment with available resources and constraints.\n", "\n", "## Expected results\n", "\n", "This project leverages approaches from the M5 Uncertainty competition, particularly in quantifying uncertainty in time series forecasting through Conformal Prediction. The focus will be on generating 9 nominal probability levels \n", - "$u \\in \\{0.005, 0.025, 0.165, 0.250, 0.500, 0.750, 0.835, 0.975, 0.995\\}$ to predict median values and construct 4 central prediction intervals (PIs) at confidence levels of $50\\%$, $67\\%$, $95\\%$, and $99\\%$. These outputs are expected to help accurately characterize the distribution's center and tails, offering a comprehensive understanding of forecast uncertainty (Makridakis et al., 2022).\n", + "$u \\in \\{0.005, 0.025, 0.165, 0.250, 0.500, 0.750, 0.835, 0.975, 0.995\\}$ to predict median values and construct four central prediction intervals (PIs) at confidence levels of $50\\%$, $67\\%$, $95\\%$, and $99\\%$. These outputs are expected to help accurately characterize the distribution's center and tails, offering a comprehensive understanding of forecast uncertainty (Makridakis et al., 2022).\n", "\n", - "However, it is important to approach the expected results with caution. Time series data is inherently complex, with varying time dependencies, different forecasting horizons, and industry-specific factors, making it challenging to generalize findings. The outcomes of this project should be taken with a grain of salt, recognizing that they may be valid within the specific context of the data used but may not be directly applicable to other datasets or domains without careful consideration.\n", + "However, it is important to approach the expected results with caution. Time series data is inherently complex, with varying time dependencies, different forecasting horizons and industry-specific factors, making it challenging to generalize findings. The outcomes of this project should be taken with a grain of salt, recognizing that they may be valid within the specific context of the data used but may not be directly applicable to other datasets or domains without careful consideration.\n", "\n", - "This does not mean that the project is without value. On the contrary, it serves as a foundation for further exploration and research in the field of Uncertainity Quantification in time series setting. By aligning with key future directions from the M5 competition—such as further developing Machine Learning methods for forecasting and enhancing reproducibility and practical implementation—the project aims to incentivize additional work in this area. Moreover, it seeks to raise awareness among academics and practitioners about the importance of understanding and communicating the full extent of uncertainty in forecasting, particularly in light of unpredictable events like COVID-19, which can lead to significant and unforeseen risks.\n", + "This does not mean that the project is without value. On the contrary, it serves as a foundation for further exploration and research in the field of uncertainity quantification in time series setting. By aligning with key future directions from the M5 competition such as further developing Machine Learning methods for forecasting and enhancing reproducibility and practical implementation, the project aims to incentivize additional work in this area. Moreover, it seeks to raise awareness among academics and practitioners about the importance of understanding and communicating the full extent of uncertainty in forecasting, particularly in scenarios such as predicting rainfall levels during flood-prone seasons, estimating energy demand during sudden weather changes, or forecasting supply chain disruptions. These situations inherently involve uncertainty and can lead to significant, unforeseen risks if not properly accounted for.\n", "\n", "## Evaluation\n", "\n", @@ -212,12 +210,12 @@ "The Weighted Scaled Pinball Loss (WSPL) will then aggregate SPL across all the time series analyzed and the nine quantiles to rank the performance of the methods. WSPL is computed as:\n", "\n", "$$\n", - "\\text{WSPL} = \\sum_{i=1}^{\\text{n\\_series}} \\frac{1}{\\text{n\\_series}} \\cdot \\frac{1}{9} \\sum_{j=1}^{9} \\text{SPL}(u_j)\n", + "\\text{WSPL} = \\sum_{i=1}^{\\text{n_series}} \\frac{1}{\\text{n_series}} \\cdot \\frac{1}{9} \\sum_{j=1}^{9} \\text{SPL}(u_j)\n", "$$\n", "\n", "where $\\text{n_series}$ is the number of time series analyzed, and each series is weighted equally. This approach differs from the WSPL proposed in the M5 competition, which weights each series based on recent actual sales. While equal weighting might be further refined in future work, it provides an effective starting point.\n", "\n", - "To evaluate if the aims of the project are achieved, the `ACI` and `EnbPI` methods will be benchmarked, and their WSPL scores compared against traditional methods such as Naive, Seasonal Naive (sNaive), Simple Exponential Smoothing (SES), Exponential Smoothing (ES), AutoRegressive Integrated Moving Average (ARIMA), and Kernel density estimates. Although this choice mirrors the M5 competition and may be subject to debate, the author believes it highlights the advantages of Conformal Prediction, particularly as traditional benchmarks, which are widely used, often assume normally distributed forecast errors, an assumption rarely met in practice." + "To evaluate if the aims of the project are achieved,`ACI`, `EnbPI` and related base methods will be benchmarked, and their WSPL scores compared against traditional methods such as Naive, Seasonal Naive (sNaive), Simple Exponential Smoothing (SES), Exponential Smoothing (ES), AutoRegressive Integrated Moving Average (ARIMA) and Kernel density estimates. Although this choice mirrors the M5 competition and may be subject to debate, it adds value due to the wide use of these traditional methods and facilitates comparison by highlighting how Conformal Prediction methods perform relative to benchmarks that often assume normally distributed forecast errors, an assumption that is infrequently met in practice." ] }, { @@ -244,7 +242,6 @@ "12. Isaac Gibbs and Emmanuel Cand`es. Adaptive conformal inference under distribution shift. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=6vaActvpcp3.\n", "13. Yang, Z., Candès, E., & Lei, L., 2024. Bellman Conformal Inference: Calibrating Prediction Intervals For Time Series. arXiv [online]. Available at: https://doi.org/10.48550/arXiv.2402.05203\n", "14. Manokhin, V., 2023. Practical Guide to Applied Conformal Prediction in Python: Learn and apply the best uncertainty frameworks to your industry applications. 1st ed. Packt Publishing.\n", - "15. Kowalczewski, J. 2019. Normalized conformal prediction for time series data.\n", "16. Shafer, G. and Vovk, V., 2008. A tutorial on conformal prediction. Journal of Machine Learning Research, 9, pp.371-421.\n", "17. Kath, C., and F. Ziel. 2021. Conformal prediction interval estimation and applications to day-ahead and intraday power markets. International Journal of Forecasting 37 (2):777–99. doi:10.1016/j.ijforecast.2020.09.006.\n", "18. Wisniewski, W., D. Lindsay, and S. Lindsay. 2020. Application of conformal prediction interval estimations to mar-\n", -- GitLab