Informed Prior Initialization for Bayesian Convolutional Neural Networks Through Deterministic Pre-Training

Abstract

Bayesian neural networks (BNNs) offer a principled framework for uncertainty quantification by treating model parameters as stochastic variables rather than fixed values. However, their practical performance is highly sensitive to prior specification. The conventional choice of uninformative zeromean isotropic Gaussian priors often leads to poor weight initialization, and suboptimal predictive performance. In this work, we introduce a two-phase training strategy designed to mitigate the limitations of uninformed priors in Bayesian convolutional neural networks (BCNNs). In the first phase, a deterministic CNN is trained to convergence using standard optimization techniques, yielding well-optimized parameters. In the second phase, these parameters are used as informative prior means when initializing the BCNN, replacing the traditional zero-mean assumption. Then the Bayesian fine-tuning leverages the reparameterization trick used in the Bayes-by-Backprop method, with predictive distributions estimated through Monte Carlo sampling. Experiments on the MNIST and Fashion-MNIST datasets demonstrate that our approach achieves lower negative log-likelihood, improved calibration, and enhanced out-of-distribution detection compared to both deterministic baselines and BCNNs with uninformed priors, all while maintaining computational efficiency close to that of standard training. Our results highlight the importance of informed priors in Bayesian deep learning and provide a practical pathway toward more reliable uncertainty estimation in convolutional architectures.