Generative Adversarial Networks

Introduction

Unless you have been living under a hut for the last year, everyone in Deep Learning — and even some not involved in Deep Learning — has heard and talked about GANs. GANs or Generative Adversarial Networks are Deep Neural Networks that are generative data models. This means that given a set of training data, GANs can learn to estimate the underlying probability distribution of the data. This is very useful because, apart from other things, we can now generate samples from the learned probability distribution that may not be present in the original training set.

What is GAN?

Generative adversarial networks (GANs) are an exciting recent innovation in machine learning. It was first introduced by Ian Godfellow in his paper Generative Adversarial Networks.

GANs are generative models: after given some training data, they can create new data instances that look like your training data. For example, GANs can create images that look like photographs of human faces, even though the faces don’t belong to any real person.

For a great example of GAN you can visit https://www.thispersondoesnotexist.com/ which was created by Nvidia. It generates a high quality image of a person who does not even exist.

Why learn GANs?

Generative Adversarial Networks (GANs) are a prominent branch of Machine learning research today. As deep neural networks require a lot of data to train on, they perform poorly if data provided is not sufficient. GANs can overcome this problem by generating new and real data, without using the tricks like data augmentation.

GANs’ potential for both good and evil is enormous, because they can learn to mimic any data distribution. That is, GANs can be taught to create worlds eerily similar to ours in any domain: images, music, speech, prose. They are robot artists in a sense, and their output is impressive – poignant even. But they can also be used to generate fake media content and are the technology underpinning Deepfakes. As shown in the image above, this GAN-generated art is sold for $433k!

How GANs work?

The Game of Probabilities

Generating new data is a game of probabilities. When we are observing the world around us and collecting data, we are experimenting. A simple example is taking a photo of a celebrity's face.

This can be considered a probabilistic experiment with an unknown outcome X, also called a random variable.

If the experiment is repeated many times, we usually define the probability that the random variable X will get the value little x, as the fraction of times that little x occurs.

All possible outcomes of such experiments build the so-called sample space, denoted $Ω$ (all possible celebrity faces).

Therefore we can consider probability as a function that takes an outcome, i.e., an element from the sample space (a photo), and maps the result to a non-negative real number so that the sum of all these numbers equals 1.

We also call this a probability distribution function P(X). When we know the sample space (all possible celebrity faces) and the probability distribution (the probability of occurrence of each face), we have the full description of the experiment and can reason about uncertainty.

Discriminator and Generator

Discriminative

Given inputs, we want to build a model that can classify the inputs to the corresponding targets as correctly as possible.

It learns the conditional probability distribution.

$p(y|x)$ - the probability of y given x should be maximum.

So the model learns to predict the labels from the data other words, it understands the decision boundary between classes.

It does not care about how the training data is generated/distributed.

Generative

Given inputs, we want to build a model that can understand the inputs to generate similar inputs and their labels from the targets.

It learns the joint probability distribution.

$P(x,y) = p(x|y).p(y)$

The model has to learn the $p(x|y)$. It cares about how the training data is generated/distributed.

GAN's Concept

Let's start with a basic idea of generating data x.

GAN composes of two deep networks, the generator and the discriminator. We will first examine how a generator creates images before learning how to train it.

First, we sample some noise $z$ using a normal or uniform distribution. With $z$ as an input, we use a generator $G$ to create an image $x$ $(x=G(z))$.

Conceptually, $z$ represents the latent features of the images generated, for example, the color and the shape. In Deep learning classification, we don't control the features the model is learning. Similarly, in GAN, we don't control the semantic meaning of $z$. We let the training process to learn it. i.e., we do not control which byte in $z$ determines the color of the hair. To discover its meaning, the most effective way is to plot the generated images and examine ourselves. The following images are generated by progressive GAN using random noise $z$!

We can gradually change one particular dimension in $z$ and visualize its semantic meaning.

So what is this magic generator G? The following is the DCGAN, one of the most popular designs for the generator network. It performs multiple transposed convolutions to upsample $z$ to generate the image $x$. We can view it as the deep learning classifier in the reverse direction.

But a generator alone will create random noise. Conceptually, the discriminator in GAN provides guidance to the generator on what images to create. By training with real images and generated images, GAN builds a discriminator to learn what features make images real. Then the same discriminator will provide feedback to the generator.

So, how is it done technically? The discriminator looks at real and generated images (training samples) separately. It distinguishes whether the input image to the discriminator is real or generated. The output $D(X)$ is the probability that the input $x$ is real, i.e. $P(class of input = real image)$.

We train the discriminator just like a deep network classifier.

• If the input is real, we want $D(x)=1$.
• If generated, we want $D(x)=0$.

Through this process, the discriminator identifies features that contribute to real images.

On the other hand, we want the generator to create images with $D(x) = 1$. So we can train the generator by backpropagating this target value back to the generator, i.e., we train the generator to create images towards what the discriminator thinks is real.

GAN’s Training Process Step by Step

1. We take some noise from a random distribution, then feed it to the Generator $G$ to produce the fake $x$ (label $y=0$) → ($x,y$) input-label pair.

2. We take this fake pair and the real pair $x$ (label $y =1$) and feed them to the Discriminator $D$.

3. The discriminator $D$ is a binary classification neural network, so it calculates the loss for both fake $x$ and real $x$ and combines them as the final loss as $D$ loss.

4. The Generator $G$ also calculates the loss from its noise as $G$ loss since each network has a different objective function.

5. The two losses go back to their respective networks to learn from the loss (adjusting the parameters with respect to the loss)

6. Apply any optimization algorithm (Grad descent, ADAM, RMS prop, etc..)

Repeat this process for a certain no of epochs or as long as you wish.

Loss Functions

Due to recent contributions to GANs, many objective functions have been introduced. Here is a summary of them.

Here we only explain the loss function that was introduced in the original paper.

We would go through how the Loss function for a GAN breaks down. The Loss term for a GAN is based on that familiar Cross-Entropy Loss, which will now be defined between the True and the Generated distributions.

• Discriminator's Loss: We're first going to consider the Loss from the perspective of the Discriminator. We want to maximize the probability that the data is identified as fake. So to break this down, $G(z)$ defines the Generator's output, and so $D(G(z))$ is the Discriminator's estimate of the probability that a fake instance is actually fake. $D(x)$ is the Discriminator's estimate of the probability that a real instance is fake, so $(1−D(x))$ is its probability estimate that a real instance is real. So together, from the point of view of the Discriminator, we want to maximize this probability.

• Generator's Loss: Remember that the Generator is taking random noise and generating an instance. It cannot directly affect the term $D(x)$, which shows up in the Loss. Because $D(x)$ is solely based on the Discriminator's operation on the real data. So for the Generator, the Generator will have the adversarial objective to the Discriminator, which means it will try to minimize this term—effectively minimizing the probability that the Discriminator can distinguish its generated data as fake ($D(G(z))$). The goal for the Generator is to minimize the $D(G(z))$ term of the objective. So the Generator's objective is to try to synthesize fake instances that fool the Discriminator. And eventually, throughout training the Discriminator, the Discriminator will be as best as possible (discriminating real vs. fake). Therefore the ultimate goal of the Generator is to synthesize fake instances that fool the best Discriminator. And this is all put together in this "Min-Max" objective function which has these two components optimized adversarially. Then after training, we can use the Generator network, which is now fully trained to produce new data instances that have never been seen before.

Different Types of Generative Adversarial Networks GANs

Here we shortly explain three of the most used GAN architectures.

1. DCGAN: DCGAN or Deep Convolutional GAN is one of the most famous implementations of GAN. It makes use of ConvNets instead of Multi-layered perceptron. Contents use a convolutional stride and are built without max pooling. Further, layers in ConvNets are not entirely connected.

2. Conditional and Unconditional GAN: It is defined as a deep learning neural network having extra parameters. In a conditional GAN (cGAN), the network iterates (during training) over all the classes that we wish to synthesize. The generator $G$ gets both a class id $c$ and a random noise vector $z$ as input, and the discriminator $D$ also gets the class id and needs to determine if its input is a real member of the given class.

3. Info GAN: it is one of the latest and advanced versions of generative adversarial networks used for unsupervised machine learning. The discriminator in an InfoGAN does not have access to the class id but must infer it from the samples given.

There are many more different types of GANs with other purposes.

You can look at The GAN Zoo, which is a repository of the list of all GANs.

GAN Applications

Image-to-Image Translation

• pix2pix: Given paired training images, the original pix2pix system learns how to turn sketches into photos, semantic maps to images, and other pixel remapping tasks

• CycleGAN: CycleGAN does not require paired training images, just collections from different sources, such as paintings and photographs of horses and zebras.

• StarGAN: StarGAN is an image-to-image translation from one domain to another. For example, given a happy face, we want to transform it into a fearful look.

• SRGAN: Create super-resolution images from the lower resolution. This is one area where GAN shows impressive results with immediate commercial possibility.

High-Resolution Image Synthesis

• GauGAN: GauGAN synthesizes photorealistic images given a semantic input layout.

• StyleGAN2: StyleGAN2 generates high-resolution images.

Image Editing

• **IcGAN:* Reconstruct or edit images with specific attributes.

GANs Issues

So, why are GANs so hard to train? As stated earlier, GANs are very hard to train in their vanilla form. We'll briefly look at why this is the case.

Hard-to-Reach Nash Equilibrium

Since these two networks shoot information at each other, it could be portrayed as a game where one guesses if the input is real or not.

The GAN framework is a non-convex, two-player, non-cooperative game with continuous, high-dimensional parameters each player wants to minimize its cost function. The optimum of this process takes the name of Nash Equilibrium - where each player will not perform any better by changing a strategy, given that the other player doesn't change their approach.

However, GANs are typically trained using gradient-descent techniques designed to find the low value of a cost function and not the Nash Equilibrium of a game.

Mode Collapse

Most data distributions are multi-modal. Take the MNIST dataset: there are 10 "modes" of data, referring to the different digits between 0 and 9.

A good generative model would be able to produce samples with sufficient variability, thus being able to generate samples from all the different classes.

However, this does not always happen.

Let's say the generator becomes good at producing the digit "3". The discriminator will likely assign high probabilities if the produced samples are convincing enough.

As a result, the generator will be pushed towards producing samples from that specific mode, ignoring the other classes most of the time. It will essentially spam the same number, and with each number passing the discriminator, this behavior will be enforced further.

Diminishing Gradient

Similar to the previous example, the discriminator may be too successful in distinguishing data samples. When that is true, the generator gradient vanishes. It starts learning less and less, failing to converge. This imbalance, the same as the previous one, can be caused if we train the networks separately. Neural network evolution can be pretty unpredictable, leading to one being ahead of the other by a mile. We primarily ensure these things don't happen if we train them together.

Implementing a GAN

Here we implement a GAN to create Anime faces.

Install libraries

Download Dataset

We download the dataset from Kaggle. A username and API Token are needed.

Imports

Config

Prepare Dataset

Generator Model

Discriminator Model

Training Config

Training Loop

Save Models

Plot Generator's & Discriminator's Loss

Generated vs. Real

Visualize GAN's Progress

Exploring the Latent Space

Conclusion

Since the publication of the original GAN paper, the number of extensions, applications, and follow-on papers has exploded. The GAN Zoo website 61 lists over 500 GAN papers published between 2014 and mid-2018, which stopped being updated. A large number of articles continue to appear each year in vision, machine learning, and graphics conferences.

Some of the more important papers since 2017 include Wasserstein GANs (Arjovsky, Chintala, and Bottou 2017), Progressive GANs (Karras, Aila, et al. 2018), UNIT (Liu, Breuel, and Kautz 2017), and MUNIT (Huang, Liu, et al. 2018), spectral normalization (Miyato, Kataoka, et al. 2018), SAGAN (Zhang, Goodfellow, et al. 2019), BigGAN (Brock, Donahue, and Simonyan 2019), StarGAN (Choi, Choi, et al. 2018) and StyleGAN (Karras, Laine, and Aila 2019) and follow-on papers (Choi, Uh et al. 2020; Karras, Laine, et al. 2020; Viazovetskyi, Ivashkin, and Kashin 2020), SPADE (Park, Liu, et al. 2019), GANSpace (H¨ark¨onen, Hertzmann et al. 2020), and VQGAN (Esser, Rombach, and Ommer 2020). You can find more detailed explanations and references to many more papers in the lectures by Johnson (2020, Lecture 20), tutorials by Goodfellow, Isola et al. (2018), and review articles by Creswell, White et al. (2018), Pan, Yu et al. (2019), and Tewari, Fried et al. (2020).

In summary, generative adversarial networks and their various extensions continue to be an extremely vibrant and functional research area, with applications such as image super-resolution, photorealistic image synthesis, image-to-image translation, and interactive image editing.

References

1. Generative Adversarial Networks, by Ian J. Goodfellow.
2. GAN Objective Functions: GANs and Their Variations
3. What is a Generative Adversarial Network?
4. A deep understanding of Generative Adversarial Networks (GANs)
5. Generative Adversarial Networks (GAN’s) with Math.
6. GAN — What is Generative Adversarial Networks GAN?
7. Generative Adversarial Networks
8. GAN — Ways to improve GAN performance
9. A Gentle Introduction to GANs and its Loss Function!
10. GAN
11. Generative Adversarial Network (GAN) for Dummies — A Step By Step Tutorial
12. The GAN Zoo
13. How to Train a GAN? Tips and tricks to make GANs work