Title: “Free Probability for predicting the performance neural networks”.

Abstract: Gradient descent during the learning process of a neural network can be subject to many instabilities. The spectral density of the Jacobian is a key component for analyzing stability. Following the works of Pennington et al., such Jacobians are modeled using free multiplicative convolutions from Free Probability Theory (FPT).  We make the following contributions:
– theoretical: refine the metamodel of Pennington et al. thanks to the rectangular analogue of free multiplicative convolutions.
– numerical: present and benchmark a homotopy method for solving the equations of free probability.
– empirical: we show that the relevant FPT metrics computed before training are highly correlated to final test accuracies – up to 85%.