Teaching Computers to Imagine with Deep Generative Models
Sergey Tulyakov, Stéphane Lathuilière
November 19 - 26, 2019
The University of Trento, Italy
Abstract
Recent methods in computer vision can be roughly categorized as those that provide some decision given an
input image or a video. Such decision includes the number of objects in the input, their type (ie car, tree,
etc). In other words, they provide some sort of labelling capabilities. We term such methods as
discriminative. Another group of methods, termed generative models, models the distribution of inputs. Such
techniques offer generative capabilities, given some input such methods can generate an image, video, audio
or text. Moreover, these methods can be conditioned on user input offering some sort of control on what is
being generated. This control includes changing a particular attribute of an image, while keeping other
attributes unchanged, such as summer to winter, male to female, smiling to non-smiling face. For humans,
changing an attribute requires careful training, specialized software and is time consuming. Therefore, such
capabilities can be considered as a form of learned imagination. Do to the ability to “imagine” generative
techniques have been widely used in a variety of applications: image synthesis, style transfer,
image-to-image translation, video synthesis and retargeting. Such models are used to enhance discriminative
techniques with unlabelled or synthetic data, learn to reconstruct 3D when 3D labels are not available.
Focus: generative models in deep learning and their applications to image and video
manipulation, translation as well as methods capitalizing upon such models to perform discriminative tasks.
Prerequisites: this course requires an understanding of the basic building blocks of
convolutional
neural networks and machine learning theory, including standard deep learning architectures, cost functions,
activations and learning paradigms. Therefore, for PhD students that have not already worked on deep neural
networks an introductory course such as
Introduction to Deep
Learning or
Deep Learning for Image Processing is highly
recommended.