NTR organizes and hosts scientific webinars on neural networks and invites speakers from all over the world to present their recent work.
On January 18 Mikhail Belkin, University of California, San Diego, USA, presented a technical Zoom webinar on What Can We Learn from Deep Learning?
About the webinar:
“A model with zero training error is overfit to the training data and will typically generalize poorly” goes statistical textbook wisdom. Yet, in modern practice over-parameterized deep networks with near perfect fit on training data still show excellent test performance. This apparent contradiction points to some troubling cracks in the conceptual foundations of machine learning.
While classical analyses rely on a trade-off balancing the complexity of predictors with training error, modern models are best described by interpolation, where a predictor is chosen among functions that fit the training data exactly, according to a certain (implicit or explicit) inductive bias.
I discussed the nature of the challenge to our understanding and pointed the way forward to analyses that account for the empirically observed phenomena and shed light on modern model selection.
In particular, I showed how classical and modern models can be unified within a single “double descent” risk curve, which subsumes the classical U-shaped bias-variance trade-off curve.
Related materials:
Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation
Mikhail Belkin, Acta Numerica, Volume 30, May 2021, pp. 203 – 248, https://arxiv.org/abs/2105.14368.
Slide deck link.
Moderator and contact: NTR CEO Nick Mikhailovsky: nickm@ntrlab.com.