Spectral Initialization of Deep Networks: Aligning Input Gram Matrices With Target Kernels
Keywords:
spectral initialization, Gram matrix, target kernel, deep neural networks, kernel alignment, eigenvalue decomposition, orthogonal initialization, neural tangent kernel, optimization, representation learning.Abstract
Deep neural networks are highly sensitive to their initialization, particularly when network depth and parameter dimensionality increase. Conventional initialization methods such as Xavier, He, orthogonal, and Layer-Sequential Unit-Variance (LSUV) primarily control activation variance or singular-value behavior but do not explicitly incorporate the geometric structure of the training data. This paper proposes a Spectral Kernel Alignment Initialization (SKAI) framework that initializes the first-layer representation by aligning the input Gram matrix with a predefined target kernel. The central idea is to transform the input representation so that its spectral structure approximates a target positive-semidefinite kernel before conventional gradient-based training begins. Given an input matrix (X), its Gram matrix (G_X=XX^T) is decomposed spectrally, while a target kernel (K_T) is constructed from class similarity, radial-basis-function similarity, polynomial similarity, or a task-specific prior. A spectrally derived transformation is then obtained by solving a regularized matrix approximation problem, producing an initialized feature representation whose covariance and principal directions are better aligned with the target kernel. The proposed framework is incorporated into a deep architecture consisting of spectral preprocessing, kernel-aligned initialization, residual feature extraction, normalization, and classification layers. Experiments are proposed on MNIST, Fashion-MNIST, and CIFAR-10 using accuracy, convergence speed, training loss, Gram alignment error, condition number, and computational overhead as evaluation metrics. The proposed method is expected to provide faster convergence, improved early-stage optimization, and stronger kernel alignment than Xavier, He, orthogonal, and LSUV initialization. The study establishes a connection between spectral matrix approximation, kernel methods, and deep-network initialization.
