Title: “Some Insights from Neural Networks with Infinite Number of Parameters.”

Abstract: The theoretical analysis of infinite-width neural networks has led to many interesting practical results (choice of initialization schemes, choice of Bayesian priors etc.). However, the traditional infinite-width framework focuses on fixed depth networks and omits the large depth behavior of these models. In this talk, I will cover different topics on the infinite-width-then-infinite-depth networks and show how we can leverage some theoretical results to obtain principled algorithms. The following papers will be covered in this talk:
• On the Impact of the activation function on deep neural networks training. ICML 2019. (Joint work with A. Doucet and J. Rousseau)
• Stable ResNet. AISTATS 2021. (Joint work with E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau)