Reducing Training Complexity Through Sensitivity-Aware Linear Dependency Pruning Of Redundant Features

Abstract

Multidimensional data usually exhibit redundant and linearly dependent dimensions that increases the computational complexity and hurts the learning performance. The typical dimension reduction methods are classical projection-based ones, and they lack interpretability in terms of features. All other feature selection methods disregard the structure of the linear dependencies or are computationally expensive, resulting in repeatedly retraining the model. In this paper we present a novel framework for feature-level dimensionality reduction that explicitly addresses the higher-order relationship among input data patterns, and with respect to redundancy attribute elimination. The removal of redundant features according to sensitivity analysis method is executed in recursive way. The proposed method integrates correlation analysis, variance inflation factor computation, rank-based structural analysis into a composite redundancy score to identify multicollinearity and dependent variables. A sensitivity-based elimination procedure is used to guarantee that removing such features does not have an independent predictive degradation below a certain tolerance level. We evaluate the framework on a high-dimensional human activity recognition data set with naturally redundant sensor-based features. The experiment results show the effectiveness and efficiency of our method in speed up training process with no or even higher accuracy than competing methods. The method is interpretable, and can be effectively learned in low-resource settings with direct operation based on features. The importance of this work is to provide a scalable, performance-aware method for simplifying high-dimensional machine learning systems and thereby facilitating their quicker adoption with greater trust in the real-world, data-intensive applications.