Hybrid Indicator Saturation Machine Learning Framework for Outlier Detection and Volatility Forecasting

Abstract

Unaccounted for outlier can severely affect the stability of models and degrade the forecasting accuracy. In financial time series, an outlier in the prior unconditional mean can introduce systematic biases in forecasts and result in model misspecification, potentially distort the parameter estimates and inference. In order to address these issues, this study proposes a hybrid approach designed to detect and model outlier effectively. The methods build upon the principles of impulse indicator saturation (IIS) by incorporating unsupervised machine learning technique density-based spatial clustering of applications with noise (DBSCAN) to identify the clusters and refine the detection of outliers. The log returns series is served as the primary input for outlier detection and volatility modeling. The proposed hybrid method IIS-DBSCAN-GARCH is assessed through a Monte Carlo simulation study and subsequently applied to five Asia-Pacific stock market daily return series spanning from 20 June 1994 to 23 December 2024. The results consistently demonstrate that IIS-DBSCAN-GARCH outperforms the classical GARCH model that does not explicitly account for outlier in both simulation and empirical study. Besides effectively identify and account for extreme fluctuations, the proposed method enhances volatility estimation and improves out-of-sample forecasting accuracy. In empirical study, the proposed hybrid methods have been compared to the classical indicator saturation method and DBSCAN approach. The findings indicate that the proposed hybrid method is more accurate with respect to identifying the outliers and shows superior in forecasting the financial volatility.