Generative Adversarial Nets:Optimizations and
Functioning
Shiv Santosh J
Computer Science & Engineering
PES University
Bengaluru, India
<shivsj9755@gmail.com>
Abstract​—Deep learning methods have had striking success
over the past few years with applications involved with data
that map high-dimensional, rich sensory input to labels. But
deep ​generative models have had less impact due to several
obstacles. Generative Adversarial Networks is a deep learning
architecture that overcomes most obstacles faced by generative
models invented until now. In this paper we will discuss how
GANs turn out to be good generative models and how
modifications within and combinations with other models
bring about exceptional generative models.
Keywords— GAN, VAE, Deep learning, Generative Models,
Similarity measures
I. INTRODUCTION
GANs are estimating generative model whose goal is to
capture a data distribution. GANs have two distinct entities
work simultaneously but entail conflicting goals. The two
entities can be thought of as G(generator) and
D(discriminator). The target of G is to capture a data
distribution and D yields a probability that a sample case is
from the training data rather than a generation of G.
In case of the models G and D being multilayer perceptrons,
the entire training can be done using back propagation.
Motivation: ​Deep learning models have shown tremendous
success in the applications that are in requirement of
discriminative models. These applications usually comprise
of high-dimensional, rich sensory input to a class label. The
primary reason behind the striking success are
backpropagation and dropout algorithms, using piecewise
linear units which have a particularly well-behaved gradient.
But when it comes to generative models, deep learning has
had a relatively lesser impact. The obstacles faced are
difficulties of approximating many intacable probabilistic
computations that arise in maximum likelihood estimation
and related strategies, and due to difficulty of leveraging the
benefits of piecewise linear units in the generative context.
A new model arose proposing adversarial nets framework.-
[2] This architecture demands availability of a large
amount of data for training. Data is a very essential resource
and one does not always have access to the amount of data
demanded by some generative models. To tackle such
scenario, one solution is implementing divide-and-conquer
approach on the problem. In [2], they implemented two
seperate GAN models which accomplish different individual
Harshavardhan Biradar
Computer Science & Engineering
PES University
Bengaluru, India
<harshabiradar84@gmail.com>
tasks. Doing so, reduces the data demand for learning the
same thing if in case of a single GAN model.
[3] If in case of applications that involve generating of
values for a particular data sample but not from a data
distribution generally, some implementation has to be
present that preserves details or computes on details that is
specific to the data sample. For example, in [3] which works
on face aging using GAN, their application demanded
feature preserving of each person in the aged output of the
model. To carry out this task, they implemented
“Identity-Preserving” optimization of GAN’s latent vector.
[4] employs the same task by replacing the generator
part of a vanilla GAN model with a variational autoencoder.
VAE better captures similarities in a data space. They also
replaced element wise errors with feature wise errors. Along
with exceptional visual fidelity in the generated version,
using negligible arithmetic modifications, high-level abstract
features were achieved.
[5] employed a text to image generation GAN model that
generates an image that is described by the text that is
passed to it.
Abbreviations and Acronyms
GAN: Generative Adversarial Networks
cGAN: conditional Generative Adversarial Networks
VAE: Variational AutoEncoders
MLE: Maximum Likelihood Estimation
DCGAN: Deep convolutional Generative adversarial
nets
G,D in equations: Generator network and discriminator
network respectively
II. ADVERSARIAL NETWORKS
Implementation of GAN model is the most unambiguous
and clear-cut when both the discriminator and generator are
multilayer perceptrons.
min max V (D, G) = E x~pdata (x) [logD(x)] + E z~pz (z) [log(1 − D(G)(z))]
G D
The objective is to learn the generator’s distribution pg over
data x . A prior is defined on input noise variables pz (z) ,
then a mapping to the data space as G(z; θg ) . θg are the
parameters to G. D(x; θd ) outputs a single scalar, probability
that some input x is from the data but not pg . Hence
explaining the equation. D is trained to maximize the
probability of assigning correct label and D to minimize
[log(1 − D(G)(z))] .
Optimizing D to completion in the inner loop is not
computationally feasible. To overcome this task the
algorithm implemented involves a pre defined set of steps ​k,​
to optimize D for every single step optimization of G.
III. APPROACH METHODS
[2] Two GAN models work together to overcome the
difficulty of less data availability. The goal of the model as a
whole is to emulate the results of a single GAN model with
large data availability with two GAN models with less data.
In the model incorporated in [2], GAN1 generates an image
containing only object contours , and GAN2 paints the
black-and-white image in order to produce the final image.
Accordingly it is understood that the output of GAN1 is the
input of GAN2. The approach is based on two factors. The
first factor is that shape, which in their opinion was the most
important element to convey the identity of an object among
the three visual features in content-based image processing
in contrary to color, texture or size.
Considering x as the original image, and z as the output
from GAN1 and y as the output of GAN2. GAN2 aims to
map random noise z and x (generated by GAN1) to y .
(x, y ) pairs are used to train the discriminator. Expecting the
discriminator to assign a positive label to true pairs and
negative label to generated pairs, (G2 (z i , xi ), xi ) , we get the
discriminator loss function as follows:
n n
1
LD =− 2n
( ∑ log(D2 (y i , xi ) + ∑ log(1 − D2 (G2 (z i , xi ), xi )))
i=1 i=1
LD is the discriminator loss function averaged over n
samples.
The loss function G2 is to maximize the probability
assigned by the discriminator to samples which are
generated.
n reconstructions and secondly, focus is on unnecessary
1
LG =− n
∑ logD2 (G2(xi , z i ), xi )
i=1
As a whole the objective function of the GAN model
incorporated can be expressed as
min max V CGAN (D2 , G2 ) = E x,y~p (x,y) [logD2 (y, x)] G2 D2
E x~p (x),z~p(z) [log(1 − D2 (G2 (x, z ), x))]
[3] Prototyping and Modelling are the two approaches
available today implemented by face aging technologies.
Prototyping approach is by estimating average age in a
predefined age group. These capture aging patterns which
are in turn applied to input faces. This method is simple and
fast but the drawback is that it does not retain input specific
details for the output it produces. In modelling approach,
employ parametric models which simulate aging
mechanisms of muscles, skin and skull of a particular
individual. But to achieve this, the model needs to train on
images of people across a wide range of ages. This is a very
costly task.
The approach incorporated in [3] has two separate
entities. (1)Age-cGAN to generate quality synthetic images
within age categories and (2)novel latent vector optimization
that enables reconstruction of images with preserving
person’s identity from input face.
The model is in high-level is a two step process. (1)
Given an input image x and age y 0 , find an optimal vector
z * which allows us to generate a reconstructed face
x̃ = G(z * , y 0 ) as close as possible to the initial one.
(2) Given the target age y target , generate the resulting face
image xtarget = G(z * , y target ) by simply switching the age at
the input of the generator.
When there is a demand for generation with certain
attributes (or conditions), cGANs(conditional GAN) are
used.
As a function of parameters of G and D, θG and θD , we
can express training of cGAN as an optimization of the
function V (θG , θD ) .
min max V (θG , θD ) = E x,y~pdata [logD (x, y )]
The problem with straight forward pixelwise Euclidean
distance between original image x ​and the reconstructed
image x̃ are two-fold. Firstly, it increases the blurriness of
details such as background, facial hair, etc which are not
relevant for identity preserving.
So the idea is minimizing difference between F R(x) and
F R(x̃) , where F R is assumed to be an ideal face recognition
+
network that recognizes a person’s identity.
 [4] For image similarity, element wise comparison is not
an ideal method. That is because it is different from how a
human would interpret images. If an image has a small
translation, it would result in a huge pixel wise error
whereas it would be nearly negligible for the human eye.
This observation with human perspective concluded that
element-wise error calculation was not a good similarity
measure. So in order to overcome this obstacle, [4]
incorporates a method where no is hand-engineered method
is used as similarity measure, but instead a function is learnt.
This is done by combining a VAE and a GAN model. The
generator of GAN and decoder of VAE are combined to one
entity.
The advantage of GAN’s discriminator is that it implicitly
learns a rich similarity metric for images, so as to
discriminate them from generated ones. This characteristic
of the GAN model was exploited so as to transfer the
properties of images learned by the discriminator into a
more abstract reconstruction error for the VAE.
The combined model is trained with a triple criterion,
Dis
ℒ = ℒprior + ℒllikel + ℒGAN , where,
ℒprior = DKL (q(z|x)||p(z)) ,
Dis
ℒllikel =− E q(z|x) [logp(Disl (x)|z)] ,
ℒGAN = log(Dis(x)) + log(1 − Dis(Gen(z)))
+ log(1 − Dis(Dec(Enc(x)))) , where,
z ~ E nc(x) = q (z|x) & x̃ ~ Dec(z) = p(x|z)
Specifically, since element-wise reconstruction errors are
not adequate for images, the VAE reconstruction error term
was replaced to a reconstruction error expressed in the GAN
discriminator. To achieve this, let Disl (x) denote the hidden
representation of the lth layer of the discriminator.
[5] With a text-to-image conversion model, one has to
overcome both contextual and perceptual losses. To ensure
good cross-modal translation, they adopted a contextual loss
term in the generator following the conditional GAN
framework. To generate realistic images, in addition
introduce perceptual loss terms for the generator,
corresponding to pixel, feature activation, and texture
reconstruction losses. Thus their approach is to regularize
the original minimax optimization for GAN with both
contextual and perceptual loss terms.
At a high level, a DCGAN is built and trained with
contextual and perceptual loss terms by conditioning on the
input text.
For contextual loss, implementation is as following ,
ιcont =− 1
m
∑ log(D(G(z (i) |h(i) ))) , where
i
h ​is for text encoding, z is for noise input and superscript
(i) denotes (i)th example.
For perceptual loss, three perceptual loss functions are
imposed, each aiming to enforce perceptual similarity
between real and generated images.
(1)Pixel reconstruction loss. Making image adjustment to
minimize pixel-wise losses is a simple approach to
encourage visual similarity between images. The pixel
reconstruction loss calculates the mean squared error
between a real image and a corresponding synthetic image.
The pixel reconstruction loss function encourages the pixels
of the two images to match. Unlike image super-resolution,
text-to-image synthesis involves one-to-many mapping
between two different data kinds. Thus high-level image
features are found more appropriate.
(2)Activation reconstruction loss. Instead of promoting
pixel-wise match between synthetic and real images,
high-level feature representations of the images could be
encouraged to be similar.
Activation outputs derived from high levels capture image
content and overall structures such as object shapes that may
be useful for classifying objects. By minimizing the
differences in the activation outputs, the generated image is
encouraged to be classified similarly as the real image,
thereby containing objects of the same class as those in the
real image.
(3) Texture reconstruction loss. Although image content and
overall structures are well captured in the activation outputs,
style-related features such as texture and recurring patterns
may not. In order to capture whether the generated image
and the real image use combinations of nearly identical set
of supporting filters, we compare Gram matrices of the
activation outputs.
IV. RESULTS
[2]
 Starting from the first row, these are the results of Two
Stage GAN, straight forward GAN architecture, and real
images.
Based on the idea that shape information plays a more
important role than color and texture for general object
identification, presented here is an adversarial approach in
order to strengthen the contribution of shape information. It
has thus been demonstrated that the use of this proposed
approach can generate better bird images than a typical
GANs model using and without using shape information on
a small-sized dataset, and the results also show that the
quality of generate images are comparable to real ones.
[3]
This shows effectiveness of synthetic aging of human
faces based on Age Conditional using Generative
Adversarial Network (Age-cGAN). The method is
composed of two steps: (1) input face reconstruction
requiring the solution of an optimization problem in order to
find an optimal latent approximation z ∗ , (2) and face aging
itself performed by a simple change of condition y at the
input of the generator. The cornerstone of this method is the
novel “Identity-Preserving” latent vector optimization
approach allowing to preserve the original person’s identity
in the reconstruction. This approach is universal meaning
that it can be used to preserve identity not only for face
aging but also for other face alterations (e.g. adding a beard,
sunglasses etc.)
[4]
The problems with element-wise distance metrics are
well known in the literature and many attempts have been
made at going beyond pixels – typically using
hand-engineered measures. Much in the spirit of deep
learning, we argue that the similarity measure is yet another
component which can be replaced by a learned model
capable of capturing high-level structure relevant to the data
distribution. In this work, the main contribution was an
unsupervised scheme for learning and applying such a
distance measure.
This was the first attempt at unsupervised learning of
encoder-decoder models as well as a similarity measure.
Their results show that the visual fidelity of the method
employed is competitive with GAN, which in that regard is
considered state-of-the art.
[5]
Generated bird images by with three different perceptual
loss functions: pixel (GAN-INT-CLS-Pixel), activation
(GAN-INT-CLS-VGG), and texture (GAN-INT-CLS-Gram)
reconstruction losses.
In [5], GAN-based text-to-image synthesis methods that use
both contextual and perceptual losses have been described.
The contextual loss in existing GAN literature focuses on
semantic relatedness between text and image, whereas the
proposed perceptual loss focuses on the object-specific
structure.
V. CONCLUSION
GANs comes with advantages and disadvantages relative
to previous modeling frameworks. The disadvantages are
primarily that there is no explicit representation of data
distribution of generator, and that D must be synchronized
well with G during training (in particular, G must not be
trained too much without updating D, much as the negative
chains of a Boltzmann machine must be kept up to date
between learning steps. The advantages are that Markov
chains are never needed, only backprop is used to obtain
gradients, no inference is needed during learning, and a wide
variety of functions can be incorporated into the model.
The aforementioned advantages are primarily
computational. Adversarial models may also gain some
statistical advantage from the generator network not being
updated directly with data examples, but only with gradients
flowing through the discriminator. This means that
components of the input are not copied directly into the
generator’s parameters. Another advantage of adversarial
networks is that they can represent very sharp, even
degenerate distributions, while methods based on Markov
chains require that the distribution be somewhat blurry in
order for the chains to be able to mix between modes.

REFERENCES

[1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,

S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,”
in Proceedings of NIPS, 2014, pp. 803–806.

[2] SYNTHESIS OF IMAGES BY TWO-STAGE GENERATIVE

ADVERSARIAL NETWORKS. Huang, Qiang, Jackson, Philip,
Plumbley, Mark D. and Wang, Wenwu (2018) In: 2018 IEEE
International Conference on Acoustics, Speech and Signal Processing,
15–20 Apr 2018, Calgary, Alberta, Canada.

[3] FACE AGING WITH CONDITIONAL GENERATIVE

ADVERSARIAL NETWORKS. Grigory Antipov, Moez Baccouche,
Jean-Luc Dugelay. arXiv:1702.01983v2 [cs.CV] 30 May 2017

[4] AUTOENCODING BEYOND PIXELS USING A LEARNED

SIMILARITY METRIC. Anders Boesen Lindbo Larsen, Søren Kaae
Sønderby, Hugo Larochelle, Ole Winther.arXiv:1512.09300v2
[cs.LG] 10 Feb 2016

[5] ADVERSARIAL NETS WITH PERCEPTUAL LOSSES FOR

TEXT-TO-IMAGE SYNTHESIS. Miriam Cha, Youngjune Gwon, H.
T. Kung. arXiv:1708.09321v1 [cs.CV] 30 Aug 2017. ACCEPTED
TO 2017 IEEE INTERNATIONAL WORKSHOP ON MACHINE
LEARNING FOR SIGNAL PROCESSING