Abstracts > Molina Mario

Joint work with Jorge Gonzalez Cazares.

Stochastic gradient descent (SGD) is among the most widely used algorithms for solving optimization problems in machine learning. Classical convergence results for stochastic approximation establish almost sure convergence under suitable regularity and moment assumptions. In particular, such results often require the stochastic gradients to have finite second moments. Nevertheless, this assumption may be restrictive in modern machine learning applications. Indeed, it has been observed empirically that gradient noise can exhibit heavy-tailed behavior, so that the distribution of stochastic gradients may fail to have a finite second moment. To address the lack of finite second moments in stochastic gradients, several robust variants of SGD have been proposed, including clipped SGD. However, a complete theoretical understanding of these schemes remains challenging. In this work, we study a robust variant of SGD in which the empirical mean used to aggregate stochastic gradients is replaced by the Median-of-Means estimator and we analyze the convergence of the resulting algorithm.