January 2025
Robust filtering and replacement of missing data to enhance prediction in linear state-space models
Dobrislav Dobrev and Paweł J. Scherzen
Abstract:
By replacing defective measurements with missing values, we can suppress distortions caused by outliers in state-space inference. Therefore, we proposed two complementary methods for enhanced outlier robust filtering and prediction. They are supervised missing data replacement (MD) when the Huber threshold is exceeded and unsupervised missing data replacement with exogenous randomization (RMDX).
Our supervised method MD outperforms existing Hoover-based linear filters, which are known to lose optimality when outliers of the same sign are clustered in time rather than arriving independently. designed to improve. The unsupervised method RMDX further aims to suppress small outliers whose size may be below the Huber detection threshold. To achieve this objective, RMDX averages filtered or predicted targets based on a measurement series containing a randomly induced subset of missing data with an exogenously set randomization rate. I will. This results in a tradeoff between regularization and bias variance as a function of the missing data randomization rate, which can be optimally set using standard cross-validation techniques.
Through Monte Carlo simulations, we verify that both methods of missing data replacement can significantly improve robust filtering, especially when combined. As further empirical validation, we document consistently attractive performance in linear models for predicting inflation trends where measurement outliers tend to cluster.
Keywords: Kalman filter, outliers, Whobelization, missing data, randomization
DOI: https://doi.org/10.17016/FEDS.2025.001
PDF: Full paper
Return to top
Last updated: January 3, 2025
