Speaker: Valentin Leplat (SkolTech, Moscow, Russia)
Webpage: https://sites.google.com/view/valentinleplat/
Date: February, 9th 2024, 14h-15h
Title: Introduction to Deep Nonnegative Matrix Factorization and Stochastic Optimization with heavy-tails
Abstract:
Part 1: Deep Nonnegative Matrix Factorization with β-Divergences
Our first topic revolves around the Deep Nonnegative Matrix Factorization (deep NMF), a novel and promising facet of unsupervised learning. Deep NMF has emerged as a potent technique for extracting multi-layered features spanning various scales. However, conventional deep NMF models have primarily relied on the least squares error as their evaluation metric, which may not be the most suitable gauge for assessing the quality of approximations across diverse datasets. For data types such as audio signals and documents, β-divergences have gained recognition as a more fitting alternative. In this seminar, we present new models and algorithms that harness β-divergences to enhance deep NMF, with an emphasis on the notion of identifiability.
Part 2: Heavy-Tailed Stochastic Optimization for Deep Neural Networks
Our second topic concerns stochastic optimization, with a particular focus on recent discoveries concerning the nature of stochastic gradient noise in deep neural network training. Contrary to the conventional assumption of Gaussian noise, empirical evidences show that gradient noise often exhibits heavy-tailed characteristics. We introduce an efficient mechanism for optimizers to handle this noise behavior. Additionally, we showcase an extension of our recently introduced stochastic optimizer, referred to as NAG-GS, specifically tailored for the training of Vision Transformers.
Click here to join the meeting: https://teams.microsoft.com/l/meetup-join/19%3aaa79c15ac331466aa8ad98cbecb29ab2%40thread.tacv2/1707130295999?context=%7b%22Tid%22%3a%22158716cf-46b9-48ca-8c49-c7bb67e575f3%22%2c%22Oid%22%3a%22c4a8aea2-7ce5-4ee9-b6c5-9fee62ad0257%22%7d