4TU.AMI Session: Mathematical aspects of Machine Learning

Organisers: Barry Koren (TU Eindhoven), Wil Schilders (TU Eindhoven)

Anastasia Borovykh (CWI)

Understanding generalisation in noisy time series forecasting

In this presentation we study the loss surface of neural networks for noisy time series forecasting. In extrapolation problems for noisy time series, neural networks, due to overparametrization, tend to overfit and the behavior of the model on the training data does not measure accurately the behaviour on unseen data due to e.g. changing underlying factors in the time series. Avoiding overfitting and finding a pattern in the data that persists for a longer period of time can thus be very challenging. In this talk we quantify what the neural network has learned using the structure of the loss surface of multi-layer neural networks. We discuss how to use the learning algorithm to control the trade-off between the complexity of the learned function and the ability of the function to fit the data. Furthermore, we gain insight into which minima are able to generalise well based on the spectrum of the Hessian matrix and the smoothness of the learned function with respect to the input.

Jim Portegies (TU Eindhoven)

Can Variational Autoencoders capture topological properties?

Although Variational Autoencoders (VAE) are often used to identify latent variables underlying certain datasets, there are many open questions about their performance.

We have investigated a particular question, namely in how far a VAE can capture topological and geometrical properties of datasets. Since there are obstructions hindering a standard VAE in capturing these properties, we designed an algorithm, the Diffusion VAE, to remove these obstructions for particular datasets by allowing for an arbitrary manifold as latent space. This is joint work with Luis Pérez and Vlado Menkovski.

Christoph Brune (University of Twente)

Deep Inversion – Autoencoders for learned regularization of inverse problems

This talk will highlight how deep learning, inverse problems theory and the calculus of variations can profit from each other. Data-driven deep learning methods have revolutionized many application fields in imaging and data science. Recently, first classical methods from the calculus of variations and inverse problems have been combined with deep learning to effectively estimate hidden parameters. Such variational networks with learned regularization and unrolled gradient flow optimization have enabled deep convolutional neural networks (CNN) to tackle inversion tasks with strongly improved performance.

However, even in the context of very basic CNN inversion methods, one fundamental aspect of inverse problems theory is still largely missing: understandable regularization scales addressing ill-posedness, i.e. stability properties of the learned inversion process. In machine learning theory this is often referred to as adversarial attacks. In this talk, we present a latent space analysis of autoencoding networks to learn the regularization of inverse problems in a controlled way. This offers new mathematical tools and insights for addressing the above limitation.

Basic deconvolution problems and realistic inversion in photoacoustic tomography illustrate the gain of deep autoencoding networks in inverse problems.