This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015 1

Towards Personalized Image Captioning

via Multimodal Memory Networks
Cesc Chunseong Park, Byeongchang Kim, and Gunhee Kim

Abstract—We address personalized image captioning, which generates a descriptive sentence for a user’s image, accounting for prior
knowledge such as her active vocabulary or writing style in her previous documents. As applications of personalized image captioning,
we solve two post automation tasks in social networks: hashtag prediction and post generation. The hashtag prediction predicts a list of
hashtags for an image, while the post generation creates a natural text consisting of normal words, emojis, and even hashtags. We
propose a novel personalized captioning model named Context Sequence Memory Network (CSMN). Its unique updates over existing
memory networks include (i) exploiting memory as a repository for multiple types of context information, (ii) appending previously
generated words into memory to capture long-term information, and (iii) adopting CNN memory structure to jointly represent nearby
ordered memory slots for better context understanding. For evaluation, we collect a new dataset InstaPIC-1.1M, comprising 1.1M
Instagram posts from 6.3K users. We further use the benchmark YFCC100M dataset [1] to validate the generality of our approach.
With quantitative evaluation and user studies via Amazon Mechanical Turk, we show that the three novel features of the CSMN help
enhance the performance of personalized image captioning over state-of-the-art captioning models.

Index Terms—Image captioning, personalization, memory networks, convolutional neural networks.

1 I NTRODUCTION

I MAGE captioning is a task of automatically generating a

descriptive sentence of an image [2], [3], [4], [5], [6], [7],
[8], [9], [10]. As this task is often regarded as one of the
frontier-AI problems, it has been actively studied in recent
vision and language research. It not only requires an algo-
rithm with in-depth understanding going beyond category
or attribute levels, but also to connect its interpretation with
a language model to create a natural sentence.
This work addresses personalization issues of image cap-
tioning, which have not been discussed in previous re-
search. We aim at generating a descriptive sentence for
an image, accounting for prior knowledge such as the
user’s active vocabulary or writing styles in her previous
documents. Figure 1(a) illustrates the motivation with an
Instagram post example. Given the same riverside image,
each user generates different captions according to their
own experience, thoughts, or writing styles, while focusing
on different themes such as solitude, Melbourne, and wedding,
respectively.
Potentially, personalized image captioning is applicable
to a wide range of automation services in photo-sharing so-
cial networks. For example, in Instagram or Facebook, users
tend to instantly take and share pictures as posts using their
mobile phones. One convenient function here is to auto-
matically craft hashtags or associated text description using
their own words, to expedite the completion of an image
post. Another example would be annotating photos from
personal photo lifelogs [11]. It often necessitates generating
• Cesc Chunseong Park is with the Lunit Incorporation, Seoul, Korea.
E-mail: cspark@lunit.io
• Byeongchang Kim and Gunhee Kim are with the Department of Computer
Science and Engineering & Center of Superintelligence, Seoul National
University, Seoul, Korea.
E-mail: byeongchang.kim@vision.snu.ac.kr, gunhee@snu.ac.kr
Manuscript received April 19, 2005; revised August 26, 2015.
diverse annotations for individual events while keeping the
consistency within the whole lifelogs. This difficulty can be
alleviated to some extent by the personalized captioning,
utilizing the prior knowledge about the user’s writing.
There are also other interesting potential applications, such
as generating personalized scene descriptions for visually
impaired people [12], suggesting diverse captions for an
image [13], and making a chit-chat dialogue agent that is
conditioned on the user profile information [14].
To show the usefulness of personalized image caption-
ing, we focus on two post automation tasks: hashtag pre-
diction and post generation. See an example of Instagram
post in Figure 1(b). The hashtag prediction automatically
predicts a list of hashtags for the image, while the post
generation creates a natural sentence consisting of normal
words, emojis, and even hashtags. Personalization is key
to success in these two tasks, because the text in social
networks is not a simple description of image content, but
the user’s own story and experience about the image with
his or her favorite vocabulary and expressions.
For achieving personalized image captioning tasks, we
propose a multimodal memory network model named as
context sequence memory network (CSMN). Our model is in-
spired by recent advances of neural memory networks [15],
[16], [17], which explicitly include memory components to
which neural networks read and write data for capturing
long-term information. Our major updates over previous
memory network models are three-fold.
First, we propose to use the memory as a context reposi-
tory of prior knowledge for personalized image captioning.
Since the topics of social network posts are too broad and
users’ writing styles are too diverse, it is crucial to leverage
prior knowledge about the authors or metadata around the
images. Our memory retains such multiple types of context
information to promote more focused prediction, including

0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015
Query Image
Beautiful solitude in the morning
User 1
The beautiful Melbourne, I love spring
User 2
Beautiful day for a wedding
User 3
(a)
Fig. 1. Problem statement of personalized image captioning with an Instagram example. (a) Personalized image captioning is motivated by that
different users are likely to generate different sentences for the same image, according to their own experiences, thoughts, or writing styles. (b) As
main applications, we address hashtag prediction and post generation tasks. Given a query image, the former predicts a list of hashtags, while the
latter generates a descriptive text to complete a post. We propose a versatile context sequence memory network (CSMN) model.
users’ active vocabulary and various image descriptors.
Second, we design the memory to sequentially store all
of the output words that the model generates. It leads to
three important advantages. First, it enables the model to
selectively attend, at every step, on the most informative
previous words and their combination with other context
information in the memory. Second, our model does not suf-
fer from the vanishing gradient problem. Most captioning
models are equipped with RNN-based encoders (e.g. [2], [7],
[8], [9], [10], [18]), which predict a word at every time step,
based only on current input and a single or a few hidden
states as an implicit summary of all previous histories.
Thus, RNNs and their variants often fail to capture long-
term dependencies, which could worsen if one wants to
use prior knowledge together. On the other hand, our state-
based sequence generation explicitly retains all the word
information in the memory to predict the next words. By
using teacher-forced learning [19], our model has a Markov
property at training time; predicting a previous word yt−1
has no effect on predicting a next word yt , which depends
on only the current memory state. Thus, the gradients
from the current time step prediction yt are not propagated
through the time. Third, our model can be easily parallelized
during training time, while RNNs are often tricky to be
parallelizable since they need to maintain hidden states of
the entire past time steps. On the other hand, our model
has no such hidden states for history, as the model does
not depend on the computations of the previous time steps,
and thus allow parallelization over every timestep in the
sequence.
Third, we propose to exploit a CNN to jointly represent
nearby ordered memory slots for better context understand-
ing. Original memory networks [16], [17] leverage time em-
bedding to model the memory order. Still its representation
power is low, because it cannot represent the correlations
between multiple memory slots, for which we exploit con-
2
Softmax
Query Image
CNN
Context yt
Memory
yt
Image feature User context Word output
CSMN model
User’s Active Vocabulary
Task1. Hashtag prediction
Task2. Post generation
(b)
volution layers that lead to much stronger representation
power. We will present a more in-depth justification of CNN
in section 4.3, and quantitative performance improvement in
section 5.3.
To evaluate the effectiveness of the three novel features
of the proposed CSMN, we collect a new personalized
image captioning dataset named InstaPIC-1.1M, comprising
1.1M Instagram posts from 6.3K users. Instagram is a great
source for personalized captioning, because posts mostly
include personal pictures with long hashtag lists and char-
acteristic text with a wide range of topics. For each picture
post, we consider the body text or a list of hashtags as
groundtruth captions. We also use an existing benchmark
dataset, Yahoo Flickr Creative Commons 100 Millon Dataset
(YFCC100M) [1], to validate the generality of our approach.
It comprises 100 million images and videos that have been
uploaded to Flickr between 2004 and 2014.
Our experimental results demonstrate that the aforemen-
tioned three unique features of our CSMN model indeed
improve captioning performance, especially for personaliza-
tion purposes. We also validate that our CSMN significantly
outperforms several state-of-the-art captioning models with
the decoders of RNNs or LSTMs (e.g. [8], [9], [20]). We
evaluate with quantitative language metrics (e.g. BLEU [21],
CIDEr [22], METEOR [23], and ROUGE [24]) and user
studies via Amazon Mechanical Turk.
This paper extends the preliminary work of Park et al.
[25] in a number of aspects. First, we make model updates
after thorough experimental comparisons, including that we
replace the single-layer CNN of [25] with multi-layer ones
for more expressive memory representation. Second, we
apply our model to the benchmark dataset YFCC100M [1]
to show better generalization performance of our approach.
Third, we perform thorough experiments to validate the
effects of individual building blocks of our algorithm. For
example, we present visualization examples of memory
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015
attention in section 5.6. Finally, we describe more details of
the algorithms, implementation and experimental settings,
including the formulation changes for different image de-
scriptors in section 4.2 and a minibatch sampling strategy in
section 4.4.
We summarize contribution of this work as follows.
(1) To the best of our knowledge, we propose a first
personalized image captioning approach. We introduce two
practical post automation tasks that benefit from personal-
ized captioning: post generation and hashtag prediction.
(2) We propose a novel memory network model named
CSMN for personalized captioning. The unique updates of
CSMN include (i) exploiting memory as a repository for
multiple context information, (ii) appending previously gen-
erated words into memory to capture long-term information
without, and (iii) adopting CNN memory structure to jointly
represent nearby ordered memory slots.
(3) For evaluation of personalized image captioning, we
introduce a novel Instagram dataset named InstaPIC-1.1M.
We make the code and data publicly available.
(4) With quantitative evaluation and user studies via
AMT, we demonstrate the effectiveness of three novel fea-
tures of CSMN and its performance superiority over state-
of-the-art captioning models, including [8], [9], [20].
2 R ELATED WORK
Image Captioning. In recent years, much work has been
published on image captioning, including [2], [3], [4], [5],
[6], [7], [8], [9], [10], [26], [27], [28], to name a few. Many
proposed captioning models exploit RNN-based decoders
to generate a sequence of words from encoded input image
representation. For example, long-term recurrent convolu-
tional networks [2] are one of the earliest model to use RNNs
for modeling the relations between sequential inputs and
outputs. You et al. [10] exploit semantic attention to com-
bine top-down and bottom-up strategies to extract richer
information from images, and couples it with an LSTM
decoder. Ren et al. [26] utilize an actor-critic reinforcement
learning model with a visual-semantic embedding reward
for better guidance of captioning. Venugopalan et al. [27] use
external image and semantic knowledge to describe unseen
object categories in output captions. Lu et al. [28] propose an
adaptive attention encoder-decoder framework to provide a
fallback option to the caption decoder.
Compared to such recent progress of image captioning
research, it is novel to replace an RNN-based decoder with
an explicit sequence memory. Moreover, no previous work
has tackled the personalization issue, which is the key
objective of this work. We also introduce post completion
and hashtag prediction as concrete and practical application
for image captioning.
Personalization in Vision and Language Research.
There have been many studies about personalization in
computer vision and natural language processing [29], [30],
[31], [32], [33], [34]. Especially, Denton et al. [29] develop a
CNN model that predicts hashtags from image content and
user information. However, this work does not formulate
the hashtag prediction as image captioning, and does not
address the post completion. In computer vision, Yao et
al. [32] propose a domain adaptation approach to classify
0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
3
user-specific human gestures. Almaev et al. [31] adopt a
transfer learning framework to detect person-specific facial
action unit detection. In NLP, Mirkin et al. [35] enhance
machine translation performance by exploiting personal
traits. Polozov et al. [34] generate personalized mathematical
word problem for a given tutor/student specification by
logic programming.
Compared to these papers, our problem setup is novel
in that personalization issues in image captioning have not
been discussed yet.
Neural Networks with Memory. Various memory net-
work models have been proposed to enable neural networks
to store variables and data over long timescales. Neural Tur-
ing Machines [15] use external memory to solve algorithmic
problems such as sorting and copying. Later, this architec-
ture is extended to Differential Neural Computer (DNC)
[36] to solve more complicated algorithmic problems such
as finding the shortest path and graph traversal. Weston et
al. [17] propose one of the earliest memory network models
for natural language question answering (QA), and later
Sukhbaatar et al. [16] modify the network to be trainable
in an end-to-end manner. Kumar et al. [37] and Milleret al.
[38] address language QA tasks proposing novel memory
networks such as dynamic networks with episodic memory
in [37] and key-value memory networks in [38].
Compared to previous memory networks, our CSMN
has three novel features as discussed in section 1: (i) memory
usage of multiple context types, (ii) an explicit memory
storage of previous words to capture long-term dependen-
cies, and (iii) CNN-based memory attention to fuse multiple
context types for better word prediction.
Natural Language Processing with Convolutional Neu-
ral Networks. Recently, CNN models have been popularly
applied to NLP tasks beyond computer vision tasks, includ-
ing [39], [40], [41], [42]. Kim [39] shows that single-layer
CNN models achieve the state-of-the-art performance on
multiple text classification tasks, such as sentiment analysis
and question classification. Later, this CNN architecture is
extended by combining highway connections and LSTMs
for language modeling problems [41]. Lai et al. [40] intro-
duce a recurrent CNN model for text classification, in which
the CNN constructs word representations while the recur-
rent structure capture contextual information of the text.
Gehring et al. [42] enhance machine translation performance
by using multi-layer CNN-based sequence generation mod-
els.
Although our method is partly influenced by such recent
progress of CNNs in the NLP tasks, it is novel to apply the
CNN structure to read/write operations in neural memory
networks.
3 DATASETS
We develop our personalized image captioning approach
with our newly collected InsaPIC-1.1M dataset and the
YFCC100M benchmark dataset [1]. Table 1 outlines key
statistics of the two datasets.
3.1 InstaPIC-1.1M
We make separate datasets for post completion and hashtag
prediction as follows. We first collect image posts from In-
stagram, which is one of the fastest growing photo-sharing
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015
1.0
97.63% covered (40000)
0.8 84.31% covered (60000)
0.6
CDF
0.4
0.2
0.0
0 50000 100000
word index
150000
Fig. 2. Cumulative distribution functions (CDFs) of the words for captions
and hashtags in the InstaPIC-1.1M dataset. For captions/hashtags, the
top 40K/60K most frequent words take 97.63%/84.31% of all the word
occurrences of the dataset, respectively.
TABLE 1
Statistics of InstaPIC-1.1M and YFCC100M dataset. We also show
average and median (in parentheses) values. The total unique posts
and users in InstaPIC-1.1M dataset are (1, 124, 815/6, 315) and in
YFCC100M dataset are (867, 922/11, 093).
InstaPIC-1.1M
Dataset # posts # users # posts/user
caption 721,176 4,820 149.6 (118)
hashtag 518,116 3,633 142.6 (107)
YFCC100M
caption 462,036 6,197 74.6 (40)
hashtag 434,936 5,495 79.2 (49)
social networks. As a post crawler, we use the built-in hash-
tag search function provided by Instagram APIs. We select
270 search keywords, which consist of the 10 most common
hashtags for each of the 27 general categories of Pinterest
(e.g. celebrities, design, education, food, drink, gardening,
hair, health, fitness, history, humor, decor, outdoor, illustra-
tion, quotes, product, sports, technology, travel, wedding,
tour, car, football, animal, pet, fashion and worldcup). We
use the Pinterest categories because they are well-defined
topics to obtain image posts of diverse users. We collect
3,455,021 raw posts from 17,813 users in total.
Next, we process a series of filtering, because posts are
often too short, highly noisy, or not written in English. We
first apply language filtering to include only English posts;
we exclude the posts where more than 20% of words are not
in English based on the dictionary en.us dict of PyEnchant.
We then remove the posts that embed hyperlinks in the body
text because they are likely to be advertisement. Finally, if
users have more than max(15, 0.15 × #user posts) non-
English or advertisement posts, we remove all of their posts.
Then, we apply filtering rules for the lengths of captions
and hashtags. We limit the maximum number of posts per
user to 1,000, not to make the dataset biased to a small
number of dominant users. We also limit the minimum
number of posts per user to 50, to be sufficiently large to
discover users’ writing patterns from posts. We also filter
out the posts whose lengths are too short or too long. We set
15 as the maximum post length, based on the statistics of the
dataset; we observe that lengthy posts tend to contain infor-
0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
4
mation irrelevant to the associated pictures. We set 3 as the
caption
hashtag minimum post length; too short posts are likely to include
only an exclamation (e.g. great!) or a short reply (e.g. thanks to
everyone!). We use the same rule for the hashtag dataset. We
observe that lengthy lists of more than 15 hashtags are often
too redundant (e.g. #fashionable, #fashionblog, #fashionista,
#fashionistas, #fashionlover, #fashionlovers). Finally, we ob-
tain about 721,176 posts for captions and 518,116 posts for
hashtags.
200000 3.2 YFCC100M Dataset
Yahoo Flickr Creative Commons 100 Million Dataset
(YFCC100M) [1] consists of 100 million Flickr user-uploaded
images and videos between 2004 and 2014 (i.e. 99,206,564
images and 793,436 videos from 578,268 different Flickr
users) along with their corresponding metadata including
titles, descriptions, camera types, usertags. We regard the
titles and descriptions as captions and usertags as hashtags.
We process a series of filtering similar to those used in
InstaPIC-1.1M. We first exclude the posts where more than
50% of words are not in English, and then remove all of
the users’ posts if the user has more than max(30, 0.15 ×
#user posts) non-English posts. We limit the maximum
number of posts per user to 1,000 and the minimum number
# words/post of posts per user to 30. We set 20 and 3 as the maximum
8.55 (8) and minimum post length, respectively. If both post titles
7.45 (7)
and descriptions satisfy these rules, we include them in
6.30 (5) the dataset. As a result, we obtain about 462,036 posts for
7.46 (6) captions and 1,353,498 posts for hashtags. To balance the
dataset size of captions and hashtags, we only consider
35% randomly sampled posts for hashtags, which finally
amounted to 434,936 posts.
3.3 Preprocessing
We separately build a vocabulary dictionary V for each of
the two tasks, by choosing the most frequent V words in
the dataset. Based on the statistics in Figure 2, we set V
of the InstaPIC-1.1M to 40K for post completion and 60K
for hash prediction. In the case of post completion, top 40K
most frequent words take 97.3% of all the word occurrences,
indicating a sufficient coverage for the vocabulary used on
Instagram. On the other hand, hashtags in Instagram show
extremely diverse neologism. Although 60K hashtags cover
only 84.31% of the word usage, we set 60K as the dictionary
size due to very slow increase of the cumulative distribution
function (CDF). Likewise, for YFCC100M, we set the size of
dictionary V to 40K for post completion and 60K for hashtag
prediction.
Before building the dictionary, we remove any urls,
special characters and unicodes except emojis. We then
lowercase words and change usernames to an @username
token. Once the dictionary is obtained, we remove out-of-
dictionary words from the text.
4 T HE C ONTEXT S EQUENCE M EMORY N ETWORK
Figure 3 illustrates the proposed context sequence memory
network (CSMN) model, which consists of a recurrent model
and context memory. The memory network models [16],
[17] allows us to handle very long-term dependencies by
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015 5

# Embedding
𝐖%& 𝐖o 𝐖f Softmax
yt 𝐖"# 𝐖"$ 𝐖",
I1 I2 … … I49
ct
$
𝐖%& Image feature

Update memory
I1 I2 … … I49 CNN
Convolution
ResNet
Output I1 I2 … … I49 u1 u2 … uD y1 … yt-1 yt
yt
𝐖"# Attention
u1 u2 … uD Input I1 I2 … … I49 u1 u2 … uD y1 … yt-1 yt
yt
𝐖"$ User context Image feature User context Word output
u1 u2 … uD qt
Wq Querying Update query
yt-1 yt
(a) (b)

Fig. 3. Illustration of the context sequence memory network (CSMN) model. (a) The context memory is constructed using image descriptions and
D frequent words from the query user’s previous posts (section 4.1). (b) The model generates an output word at every step t based on the memory
state, and the newly generated word is inserted into the word output memory (section 4.2).

easily reading and writing to parts of a memory component.
The input is a query image Iq of a specific user, and the
output is a sequence of words: {yt } = y1 , . . . , yT , each of
which is a symbol coming from the dictionary V . That is,
{yt } corresponds to a list of hashtags in hashtag prediction,
and a post sentence in post generation. Another input is
the context information to be added to memory, such as
active vocabulary of a query user, which will be discussed
in section 4.1.
Since both tasks can be formulated as word sequence
prediction for a given image, we exploit the same CSMN
model with only changing the dictionary. Especially, we also
regard hashtag prediction as sequence prediction instead
of prediction of a bag of orderless tag words. Although
some previous papers (e.g. [29], [43]) formulate the hashtag
prediction as a ranking problem, we believe it is more
advantageous to pose it as a sequence generation for several
reasons. First, hashtags in a post tend to have strong co-
occurrence relations, it is better to take previous hashtags
into account to predict a next one. Second, users often write
a list of hashtags according to their own habitual sequence
patterns, which can be captured by the sequence modeling.
Third, models also need to decide the number of hashtags
to be generated, according to the user’s post history or the
content of a given image, which can only be learned through
sequence modeling. It will be validated by our experimental
results.
4.1 Construction of Context Memory
We present what types of context information are to be
stored in the memory. As in Figure 3(a), we store three types
of context information: (i) image memory for representation of
a query image, (ii) user context memory for TF-IDF weighted
D frequent words from the query user’s previous posts,
and (iii) word output memory for previously generated words.
Following [16], [17], each input to the memory is embedded
into input and output memory representation, for which
we use superscript a and c, respectively. The input memory
representation is used to decide the attention weights over
memory slots for a given query image, while the output
memory representation is used to compute the memory
readout vector.
Image Memory. We represent images using ResNet-101
[44] pretrained on the ImageNet 2012 dataset. We test two
different descriptions: (7 × 7) res5c feature map, and pool5
feature vectors. The res5c feature map denoted by Ir5c ∈
R2,048×7×7 is useful if a model exploits spatial attention;
otherwise, the pool5 feature Ip5 ∈ R2,048 is used as a feature
vector of the image. Hence, the pool5 is inserted into a single
memory cell, while the res5c feature map occupies 49 cells,
on which the memory attention later can focus on different
regions of an (7 × 7) image grid. We will compare these two
descriptors in the experiments.
The image memory vector mim ∈ R1,024 for the res5c
feature is represented by
maim,j = ReLU(Wim
a r5c
Ij + baim ), (1)
mcim,j = ReLU(Wim
c
Ir5c
j + bcim ), (2)
for j = 1, . . . , 49. The parameters to be learned include
a,c
Wim ∈ R1,024×2,048 and ba,c
im ∈ R
1,024
. The ReLU indicates
an element-wise ReLU activation [45]. For the pool5, we use
a/c a/c a/c
mim,j = ReLU(Wim Ip5
j + bim ). (3)
for j = 1. In Eq.(3), we simply present two equations for
input and output memory as a single one using superscript
a/c. We below derive the formulation assuming that we use
the res5c feature, because it subsumes the pool5 feature.
User Context Memory. In a personalized setting where
the author of a query image is identifiable, we define
{ui }D
i=1 by selecting D most frequent words from the user’s
previous posts. We input {ui }Di=1 into the user context mem-
ory in a decreasing order of TF-IDF scores, in order to exploit
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015
CNN later effectively. This context memory improves the
model’s performance by focusing more on the user’s writing
style of active vocabulary or hashtags. To build {ui }Di=1 , we
compute TF-IDF scores and select top-D words for a given
user. Using TF-IDF scores means that we do not include too
general terms that many users commonly use, because they
are not helpful for personalization. Finally, the user context
a/c
memory vector mus ∈ R1,024 becomes
uaj = Wea uj , ucj = Wec uj ; yj ; j ∈ 1, . . . , D
a/c a/c
mus,j = ReLU(Wh [uj ] + bh ),
where uj is a one-hot vector for j -th active word. Parame-
a/c
ters include We ∈ R512×V and Wh ∈ R1,024×512 . We use
the same Wh for both input and output memory, while we
a/c
learn separate word embedding matrices We .
Word Output Memory. As shown in Figure 3(c), we
insert a series of previously generated words y1 , . . . , yt−1
into the word output memory, which is represented as
oaj = Wea yj , ocj = Wec yj ; j ∈ 1, . . . , t − 1
a/c a/c
mot,j = ReLU(Wh [oj ] + bh ).
where yj is a one-hot vector for j -th previous word. We use
a/c
the same word embeddings We and parameters Wh , bh
a/c
with user context memory in Eq.(4). We update mot,j for
every iteration whenever a new word is generated.
Finally, we concatenate the presentation of all memory
a/c a/c a/c a/c a/c
types: Mt = [mim,1 ⊕· · ·⊕mim,49 ⊕mus,1 ⊕· · ·⊕mus,D ⊕
a/c a/c
mot,1 ⊕ · · · ⊕ mot,t−1 ]. We use m to denote the memory
size, which is the sum of sizes of the three memory types:
m = mim + mus + mot .
4.2 State-Based Sequence Generation
RNNs and their variants have been widely used for se-
quence generation via recurrent connections throughout
time. However, our approach does not involve any RNN
module, but sequentially store all of the previously gener-
ated words into the memory. It enables the prediction of
each output word by selectively attending on the combina-
tions of all previous words, image regions, and user context.
Input word. We discuss how to predict a word yt at
time step t based on the memory state (see Figure 3(b)).
Letting one-hot vector of the previous word to yt−1 , we
first generate an input vector qt at time t to our memory
network as
qt = ReLU(Wq xt + bq ), where xt = Web yt−1 .
where Web ∈ R512×V and Wq ∈ R1,024×512 are learned.
Next qt is fed into the attention model of context memory:
pt = softmax(Mat qt ), Mot (∗, i) = pt ◦
0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Transactions on Pattern Analysis and Machine Intelligence
6
where we compute how well the input vector qt matches
with each cell of memory Mat by a matrix multiplication
followed by a softmax. That is, pt ∈ Rm indicates the
compatibility of qt over m memory cells. Another inter-
pretation is that pt indicates which part of input memory
is important for input qt at the current time step (i.e. to
which part of memory the attention turns at time t [9]). Next,
we rescale each column of the output memory presentation
Mct ∈ Rm×1,024 by element-wise multiplication (denoted
(4) by ◦) with pt ∈ Rm . As a result, we obtain the attended
output memory representation Mot , which is decomposed
(5)
a/c
into three memory types as Mot = [moim,1:49 ⊕ mus,1:D ⊕
a/c
mot,1:t−1 ].
We then apply a CNN to the attended memory output
Mot . As will be shown in our experiments, using a CNN
significantly boosts the captioning performance. This is
mainly thanks to that the CNN allows us to obtain a set of
expressive representation by fusing multiple heterogeneous
cells with different filters. In next section 4.3, we will further
justify the intuition of why memory CNNs help enhance the
captioning performance.
We test two different CNN types as follows: (i) single-
(6) layer with multiple sizes of kernels [39], and (ii) multiple-
layers with a single size of kernel [42].
(7)
Single-layer CNNs. We define a set of three filters with
a depth of 300 and window sizes h = [3, 4, 5]. We separately
apply a single convolutional layer and max-pooling layer to
each memory type. For h = [3, 4, 5],
chim,t = maxpool(ReLU(wim
h
∗ moim,1:49 + bhim )) (10)
where ∗ indicates the convolutional operation. Parame-
ters include biases bhim ∈ R49×300 and filters wim h
∈
R[3,4,5]×1,024×300 . Via max-pooling, each chim,t is reduced
from (300×[47, 46, 45]) to (300×[1, 1, 1]). Finally, we obtain
cim,t by concatenating chim,t from h = 3 to 5. We repeat
the convolution and maxpooling operation of Eq.(10) to the
other memory types as well. As a result, we obtain ct =
[cim,t ⊕cus,t ⊕cot,t ], whose dimension is 2, 700 = 3×3×300.
Multi-layer CNNs. We define a multi-layer CNN by
stacking several layers, of which each contains a convolution
followed by a non-linearity. After thorough tests, we use 3-
layer CNNs with a depth 1,024 and a window size h = 3,
although our formulation is orthogonal to any choice of
CNN structure. Each convolution filter is parameterized as
l
wim ∈ R3×1,024×(2×1,024) and blim ∈ R49×(2×1,024) . We use
the gated linear units (GLU) [46] as non-linearity, which im-
plement a simple gating mechanism over the output of the
convolution [c1 c2 ] ∈ R(2×1024) : v([c1 c2 ]) = c1 ◦ σ(c1 B) ∈
R1,024 where σ is a sigmoid and ◦ is the element-wise
multiplication. That is, the GLU activation halves the output
(8)
of convolution. We also add residual connections from the
input of each convolution to the output of the layer. As a
result, the output of the l-th layer clim becomes
clim,t = v(wim
l
∗ cl−1 l l−1
im,1:49 + bim ) + cim,t , (11)
Mct (∗, i), (9)
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015
where c1im,t is initialized as moim,1:49 . We apply the convo-
lution operation of Eq.(11) to the other memory types in the
same way. As a result, we obtain ct = [clim,t ⊕ clus,t ⊕ clot,t ],
whose dimension is 3, 072 = 3 × 1, 024.
Use of pool5 Instead of res5c Features. In the above,
we describe our model assuming that we use the res5c
features Ir5c ∈ R2,048×7×7 for image representation. Here
we discuss two formulation changes when we use the pool5
features Ip5 ∈ R2,048 .First, Eq.(10) of the output of memory
is changed to
chim,t = ReLU(wim
h
moim,1 + bhim ). (12)
where bhim ∈ R1,800 , wim h ∈ R1,024×1,800 . After experi-
ments, we find out that adding ReLU on Eq.(12) slightly
improves performance.
Second, the memory output concatenation for pool5
features is changed to ct = cim,t + [cus,t ⊕ cot,t ], where
the dimension of ct is changed from 2, 700 = 3 × 3 × 300 for
rec5c features to 1, 800 = 2 × 3 × 300 for pool5 features.
Output word. After CNNs are applied, we compute the
output word probability st ∈ RV as
ht = ReLU(Wo ct + bo ), (13)
st = softmax(Wf ht ). (14)
We obtain the hidden state ht by Eq.(13) with a weight
matrix Wo ∈ R2,700×2,700 and a bias bo ∈ R2,700 . We then
compute the output probability st over vocabulary V by a
softmax layer in Eq.(14).
Finally, we select the word that attains the highest proba-
bility yt = argmaxs∈V (st ). Unless the output word yt is the
EOS token, we repeat generating the next word by feeding
yt into the word output memory in Eq.(6) and the input of
Eq.(8) at time step t + 1. As simple post-processing, only for
hashtag prediction, we remove duplicate output hashtags.
In summary, this inference is greedy in the sense that the
model creates the best sequence by a sequential search for
the best word at each time step.
4.3 Why can Using Memory CNN be Helpful?
While conventional memory networks cannot model the
structural ordering unless time embedding is added to the
model (e.g. [16]), we propose to exploit the memory CNN
to model the structural ordering which results in a stronger
representation power. More specifically, for the word output
memory, the CNN is useful to represent the sequential order
of generated words. For the user context memory, the CNN
can correctly capture the importance order of the context
words, given that we store the user’s frequent words in
a decreasing order of TF-IDF weighted scores (rather than
putting them in a random order).
For example, as illustrated in Figure 4, suppose that two
users have active words related to art and fashion in the user
context memory, respectively. If the art-related words are
at the top of the memory, the street is joined with art, and
this user can be modeled to be interested in art; thereby the
7
art
fashion
street Street fashion of the day
User 1
camera
paint
fashion
painting
street My first street painting
Query Image
User 2
art
camera
User Context CNN
Fig. 4. An intuitive example of user context memory for showing why
the memory CNN can be helpful for better captioning. Suppose that two
users have street as their active words in the memory. User 1 has art-
related words at the top of the memory, the street is joined with art, and
the meaning of street can be interpreted similarly as street art. User
2 has fashion-related words in the memory, the same word street is
interpreted as street fashion. In summary, the CNN encourages an effect
of n-gram.
meaning of street can be interpreted similarly as street art.
If the fashion-related words are at the top of the memory,
the same word street is interpreted as a street fashion. That
is, the CNN encourages an effect of n-gram; on the other
hand, without the memory CNN, every single memory slot
is accessed separately, and thus it is difficult to distinguish
between these two different uses of street. Note that such
effect is also exploited in other types of memory.
4.4 Training
We adopt teacher-forced learning [19], in which we provide
the groundtruth words up to t − 1 to train the prediction at
t for sequence learning. We use the softmax cross-entropy
loss as the cost function, which minimizes the negative log
likelihood from the estimated yt to its corresponding target
word yGT,t at every time step. We randomly initialize all
thepparameters with a uniform unit scaling of 1.0 factor:
[± 3/dim].
We apply a mini-batch stochastic gradient descent. We
select the Adam optimizer [47] with β2 = 0.9, β2 = 0.999
and  = 1e − 08. To speed up the training procedure, we
use four GPUs for data parallelism, and set a batch size
to 200 for each GPU. We earn the best results when the
initial learning rate is set as 0.001 for all the models. At
every 5 epochs, we divide a learning rate by 1.2 to gradually
decrease it. We train our models up to 20 epochs.
Although our model can take variable-length sequences
as input, to speed up training, it is better for a minibatch
to consist of sentences with the same length. Therefore, we
randomly group training samples to a set of minibatches,
each of which has the same length if possible. We then
randomly shuffle a batch order so that short and long
minibatches are mixed. We also use a curriculum learning
proposed in [48], which empirically leads to better training.
5 E XPERIMENTS
We compare the performance of our approach with other
state-of-the-art models via quantitative measures and Ama-
zon Mechanical Turk (AMT) studies. We make public our
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015 8

(GT) the sky is the limit (GT) i love my tatoo (GT) fiery sunset over the sihouette (GT) view for the weekend (GT) pretty flowers from the (GT) dresen by the end of (GT) thanks for the love
of magnificent mt taranaki hubby 🌸💐 the day @username
(Ours) i love the view from (Ours) i love my new (Ours) not a bad view from
my apartment window #pdx bracelet thanks @username (Ours) the last of the sunset from my the office (Ours) my beautiful flowers (Ours) a postcard from a (Ours) thanks for the love
balcony from my hubby lovely past night @username
(seq2seq) the best part about (UsrIm) my beautiful little (Showtell) is a beautiful
(Usr) wooden ceiling in the arundel (NoFB) I love (seq2seq) someone my heart (UsrIm) because cartoons
the sun girl place to be in the world
castle church are important and gummen
_UNK
shouldn’t be able to hide
behind any religion

(GT) posing with the cat (GT) the path to (GT) prime ministers global (GT) christmas at the (GT) sunset in the park (GT) red the cat tucks (GT) the first sale of the season
(Ours) molly with the _UNK fellowship reception capitol (Ours) a sunset in the forest herself in to a comfy (Ours) the pick of the day
cat (Ours) the (Ours) the prime ministers reception (Ours) speaking at the (seq2seq) happy birthday to all (Ours) a cat in the new (UsrIm) in the middle of unpacking
(attendtell) the in the overgrown path to at opening reception home our whole place was a mess
_UNK the trail (UsrIm) england twenty0 cricket (UsrIm) art at the (im2txt) and the _UNK cat
(seq2seq) the view team capitol _UNK …
from our _UNK

Fig. 5. Examples of post generation from InstaPIC-1.1M (top) and YFCC100M (bottom). In each set, we present a query image, groundtruth (GT),
and generated posts by our method (Ours) and baselines. The @username shows an anonymized user. Most of the predicted texts are relevant
and meaningful for the query images.

(GT) #vsco #sunset #vscocam
#pnw
(Ours) #vscocam #sunset
#latergram #vsco
(GT) #food #foodblog #foodie
#foodporn
(Ours) #foodstagram #foodie
#foodporn #food #dinner #yum
#foodphotography #instafood …
(GT) #artisticcommunity #art
#artist #photographer …
(Ours) #art #artist #interiors
#handmade #design #interior
(GT) #dickblickartsupplies #art
#artists
(Ours) #beautiful #style #fashion
#ootd
(GT) #memories #nolimits
#vacation #family #fitfam
(Ours) #summer #kids #love
#family #memories #sunshine
#lake
(GT) #greensmoothie #dairyfree
#lifewithatoddler #glutenfree
#vegetarian …
(Ours) #greensmoothie
#greenjuice #smoothie #vegan
#raw #juicing #eatclean #detox
#cleanse

(GT) city new_york street urban
(Ours) city new_york sidewalk
street urban
(AttendTell) art new sidewalk
(GT) beautiful california canon friends
nature …
(Ours) canon dog friends
(im2txt) garden lawn outside patio
yard yellow …
(GT) arcade corridor light night shop
street …
(Ours) japan street tokyo urban
(1NN-UsrIm) barzil city cloud night
sky
(GT) bw black clothing me stripes (GT) bridges canon chicago (GT) bicycling cycling
sweater white river racing
(Ours) b&w black close-up white (Ours) canon city downtown (Ours) bike cycling
(1NN-UsrIm) acoustic ca hdr london river street downhill fontana_ca
california camera d0 diamondbar (seq2seq) beach canon coast mountaint race racing
… horizon landscape rocks … (AttendTell) bicycle bike

Fig. 6. Examples of hashtag prediction from InstaPIC-1.1M (top) and YFCC100M (bottom). In each set, we present with a query image, groundtruth
(GT), and generated posts by our method (Ours) and baselines. Most of the predicted texts are relevant and meaningful for the query images. Bold
hashtags indicate correctly predicted ones (i.e. the hashtags that appear in both groundtruth and prediction).

source code and datasets at https://github.com/cesc-park/ 5.1 Experimental Setting

attend2u.
We use the image of a test post as a query and associated
hashtags and text description as groundtruth (GT). For
evaluation metrics of hashtag prediction, we compute the

0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015
F1-score as a balanced average metric between precision and
recall: 2(1/precision+1/recall)−1 . For evaluation measures
of post generation, we compute the language similarity
between predicted sentences and GTs. We exploit BLEU [21],
CIDEr [22], METEOR [23], and ROUGE-r [24] scores. In all
measures, higher scores indicate better performance.
For InstaPIC-1.1M, we randomly split the dataset into
90% for training, 5K posts for test and the rest for validation.
For YFCC100M, we randomly split the dataset into 90% for
training, 5% for validation and 5% for test. We divide the
dataset by users so that training and test users are disjoint, in
order to measure the generalizability of methods. If user’s
posts exist both in training and test sets, then prediction can
be often trivial by simply retrieving their closest posts in the
training set.
While some benchmark datasets for image captioning
(e.g. Flickr30K [49] and MS COCO [50]) have multiple GTs
(e.g. five sentences per image in the MS COCO), our dataset
has only one GT text and hashtag list per test example.
Hence, the absolute metric values in this work may be lower
than those in these benchmark datasets.
5.2 Baselines
As baselines, we select multiple nearest neighbor ap-
proaches, one language generation algorithm, two state-of-
the-art image captioning methods, and multiple variants of
our model. As straightforward baselines, we first test the
1-nearest search by images, denoted by (1NN-Im); for a
query image, we find its closest training image using the `2
distance on ResNet pool5 descriptors, and return its text as
a prediction. Second, we test the 1-nearest search by users,
denoted by (1NN-Usr); we find the nearest user whose 60
active words overlap the most with those of the query user,
and then randomly select one post of the nearest user. The
third nearest neighbor variant denoted by (1NN-UsrIm) is
to find the 1-nearest image among the nearest user’s images,
and return its text as a prediction.
As a language-only method, we use the sequence-
to-sequence model by Vinyals et al. [20], denoted by
(seq2seq). It is a recurrent neural network with three
hidden LSTM layers, and originally applied to the language
translation. This baseline takes 60 active words of the query
user in a decreasing order of TF-IDF weights, and predicts
captions. Since this baseline does not use an image for text
generation, this comparison quantifies how important the
image is for hashtags or text prediction.
We also compare with the two state-of-the-art image
captioning methods which have no personalization. The
first baseline is (ShowTell) from [8], which is a multi-
modal CNN and LSTM model. The second baseline is
the attention-based captioning model from [9] denoted by
(AttendTell).
We compare different variants of our method (CSMN-*).
To validate the contribution of each component, we exclude
one of the key components from our model as follows:
(i) without the memory CNN in section 4.2 denoted by
(-NoCNN-), (ii) without user context memory denoted by
(-NoUC-), and (iii) without feedback of previously gen-
erated words to output memory by (-NoWO-). That is,
the (-NoCNN-) quantifies the performance improvement by
0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
9
the use of the memory CNN. The (-NoUC-) is the model
without personalization; it does not use the information
about query users, such as their D active words. Finally, the
(-NoWO-) is the model without sequential prediction. For
hashtag prediction, the (-NoWO-) indicates the performance
of separate tag generation instead of sequential prediction
of our original proposal. We test two different image de-
scriptors in section 4.1: (7 × 7) res5c feature maps and pool5
feature vectors, denoted by (-R5C) and (-P5) respectively.
We also test the multi-layer CNNs for memory as described
in section 4.2, which are denoted by (-Mul). We evaluate
the effects on the sizes of user context memory: (-W20-)
and (-W80-) or (-W100-). Finally, we test the effect of the
beam search with different beam sizes of [3,5,7], which are
denoted by (-B3), (-B5) and (-B7), respectively.
5.3 Quantitative Results
Table 2 and 3 summarize the quantitative results of post
generation and hashtag prediction, respectively. Since al-
gorithms show similar patterns in both tasks, we below
analyze the experimental results all together.
First of all, according to most metrics in both tasks,
our approach (CSMN-*) significantly outperforms baselines.
We can divide the algorithms into two groups with and
without personalization; the latter includes (ShowTell),
(AttendTell), (1NN-Im), and (CSMN-NoUC-P5), while the
former comprises the other methods. Our (CSMN-NoUC-P5)
ranks the first among the methods with no personalization,
while the (CSMN-P5) achieves the best overall.
We summarize other interesting observations as follows.
First, among the baselines, the simple nearest neighbor ap-
proach (1NN-UsrIm) turns out to be the strongest candidate.
Second, our approach becomes significantly worse, if we
remove one of the key components, such as memory CNN,
personalization, or sequential prediction. Third, among the
tested memory sizes of user context, the best performance
is obtained with 60. With larger memory sizes, attention
learning becomes harder. Moreover, we choose the size of
60 (40 for YFCC100M) based on the statistics of our dataset;
with too large a size, there are many empty slots, which also
make attention learning difficult. Fourth, interestingly, for
image description, using pool5 features occupying only a
single memory slot leads to better performance than using
(7 × 7) res5c feature maps with 49 slots. This is mainly
because attention learning quickly becomes harder with a
much larger dimension of image representation. Another
reason could be that users do not tend to discuss in the level
of details about individual (7 × 7) image grids, and thus
a holistic view of the image content is often sufficient for
prediction of users’ posts. Fifth, the beam search does not
improve the performance in both tasks. The reason of the
poor performance by the beam search might be similar to
that observed in [51], which reports that the beam search
is successful for producing generic, repetitive, high-level
image description (e.g. This is a picture of a dog), but poor for
story generation. That is, our personalized post prediction
is more similar to the subjective, specific story generation
rather than generic, high-level image captioning. Sixth,
post generation is more challenging than hashtag prediction.
This is because the expression space of post generation is
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015 10

TABLE 2 TABLE 3
Evaluation of post generation for the InstaPIC-1.1M and YFCC100M dataset. As Evaluation of the hashtag prediction. We show test
performance measures, we use language similarity metrics (BLEU, CIDEr, METEOR, results for split by users in the left and split by
ROUGE-L). The methods with [∗ ] use no personalization. posts in the right.

InstaPIC-1.1M InstaPIC-1.1M
Methods B-1 B-2 B-3 B-4 METEOR CIDEr ROUGE-L Methods F1 score
(seq2seq) [20] 0.050 0.012 0.003 0.000 0.024 0.034 0.065 (seq2seq) [20] 0.132 0.085
(ShowTell)∗ [8] 0.055 0.019 0.007 0.003 0.038 0.004 0.081 (ShowTell)∗ [8] 0.028 0.011
(AttendTell)∗ [9] 0.106 0.015 0.000 0.000 0.026 0.049 0.140 (AttendTell)∗ [9] 0.020 0.014
(1NN-Im)∗ 0.071 0.020 0.007 0.004 0.032 0.059 0.069 (1NN-Im)∗ 0.049 0.110
(1NN-Usr) 0.063 0.014 0.002 0.000 0.028 0.025 0.059 (1NN-Usr) 0.054 0.173
(1NN-UsrIm) 0.106 0.032 0.011 0.005 0.046 0.084 0.104 (1NN-UsrIm) 0.109 0.380
(CSMN-NoCNN-P5) 0.086 0.037 0.015 0.000 0.037 0.103 0.122 (CSMN-NoCNN-P5) 0.135 0.310
(CSMN-NoUC-P5)∗ 0.079 0.032 0.015 0.008 0.037 0.133 0.120 (CSMN-NoUC-P5)∗ 0.111 0.076
(CSMN-NoWO-P5) 0.090 0.040 0.016 0.006 0.037 0.119 0.116 (CSMN-NoWO-P5) 0.117 0.244
(CSMN-R5C) 0.097 0.034 0.013 0.006 0.040 0.107 0.110 (CSMN-R5C) 0.192 0.340
(CSMN-P5) 0.171 0.068 0.029 0.013 0.064 0.214 0.177 (CSMN-P5) 0.230 0.390
(CSMN-P5-Mul) 0.145 0.049 0.022 0.009 0.049 0.145 0.143 (CSMN-P5-Mul) 0.320 0.270
(CSMN-W20-P5) 0.116 0.041 0.018 0.007 0.044 0.119 0.123 (CSMN-W20-P5) 0.147 0.349
(CSMN-W100-P5) 0.109 0.037 0.015 0.007 0.042 0.109 0.112 (CSMN-W80-P5) 0.135 0.341
(CSMN-P5-B3) 0.128 0.049 0.019 0.008 0.052 0.124 0.128 (CSMN-P5-B3) 0.224 0.386
(CSMN-P5-B5) 0.132 0.051 0.019 0.008 0.054 0.123 0.130 (CSMN-P5-B5) 0.220 0.378
(CSMN-P5-B7) 0.129 0.050 0.020 0.009 0.054 0.126 0.128 (CSMN-P5-B7) 0.218 0.372
YFCC100M YFCC100M
(seq2seq) [20] 0.076 0.010 0.000 0.000 0.034 0.069 0.066 (seq2seq) [20] 0.112 0.337
(ShowTell)∗ [8] 0.027 0.003 0.000 0.000 0.024 0.003 0.043 (ShowTell)∗ [8] 0.047 0.047
(AttendTell)∗ [9] 0.088 0.010 0.001 0.000 0.034 0.076 0.116 (AttendTell)∗ [9] 0.069 0.313
(1NN-Im)∗ 0.033 0.006 0.002 0.000 0.020 0.063 0.043 (1NN-Im)∗ 0.020 0.177
(1NN-Usr) 0.032 0.003 0.001 0.000 0.016 0.028 0.041 (1NN-Usr) 0.029 0.353
(1NN-UsrIm) 0.039 0.005 0.001 0.000 0.021 0.050 0.050 (1NN-UsrIm) 0.065 0.609
(CSMN-P5) 0.106 0.034 0.012 0.004 0.033 0.064 0.099 (CSMN-P5) 0.229 0.335
(CSMN-P5-Mul) 0.116 0.036 0.010 0.003 0.036 0.060 0.111 (CSMN-P5-Mul) 0.240 0.348

#art #artist #illustration #paint
#design #artwork #painting
#motivation #love #family #joy
#peace #prayer
#love #gift #makeup #skincare #mom
#pink #love #valentinesday #engaged
#bride #wedding #bridetobe
#vintagebooks #kids #art #gift #vintage
#handmade #packaging #books #diy
#hair #natural my boys cheering on the sidelines for the game beautiful solitude in the morning
#saturdaynight #fashion golfing with my little bro ! the beautiful melbourne I love spring
#shenanigans #selfie #makeup… cubs game with the girls at the rugby today beautiful day for a wedding
#love #family my baby cousin @username and I
#pro #selfie

Fig. 7. Three examples of hashtag prediction and two examples of post prediction with query images and multiple predictions by different users
(shown in different colors). Predicted results vary according to query users, but still are relevant and meaningful for the query images.

#sew #art #stitch #artist #art #artist #interiors #sketch #art #artist #illustration #instaart #blue #sketch #art #summer #sea #sky #cute #pet #catsofinstagram #cat #cats
#sewing #illustration #handmade #design #interior #design #artwork #webstagram #drawing #artist #tree
#embroidery #etsy #bird (a) Design topic (b) Substantially different topic

Fig. 8. Six examples of hashtag prediction for a single user, whose most posts are about design. (a) For design-related query images, our CSMN
predicts relevant hashtags to the design topic. (b) For the query images of substantially different topics, our CSMN is also resilient to predict
meaningful hashtags for the images.

much larger since the posts’ text includes any combinations of hashtags generated by our model is 5.95, which is shorter
of words, emojis, and symbols. Finally, the average length than the average GT length of 8.71. It is mainly because we

0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015
adopt early stopping during training to avoid overfitting.
We choose F1 scores as the main evaluation metric, and the
F1 scores are penalized severely when the model predicts
wrong hashtags. Therefore, our model is trained to prefer
shorter hashtags compared to the GT answers.
Given that Instagram provides a function of hashtag
automation from previous posts when writing a new post,
we test another dataset split for hashtag prediction. That
is, we divide the dataset by posts so that each user’s posts
are included in both training and test sets. We call this as
split by posts, while the original split as a split by users.
Table 3 shows the results of split by users in the left and
split by posts in the right. We observe that, due to the
automation function, many posts in our training and test
set have almost identical hashtags. This setting is highly
favorable for (1NN-UsrIm), which returns the hashtags of
the closest training image of the query user. Interestingly,
our (CSMN-P5) works better than (1NN-UsrIm) even in the
setting of split by posts, although its performance margin (i.e.
0.01 in F1 score) is not as significant as in the split by users
(i.e. 0.121). While our (CSMN-P5) is the best in split by posts,
(CSMN-P5-Mul) is overall the best in split by users. Given
that, the test and training data in the split by posts are more
similar to one another than in split by users, we can conclude
that (CSMN-P5-Mul) generalizes better than (CSMN-P5).
We also summarize notable results in YFCC100M as
follows. First, compared to InstaPIC-1.1M, all methods show
lower prediction accuracies for caption generation. Its main
cause may be the way of data collection; while the InstaPIC-
1.1M is collected from 10 most common hashtags for each of
27 Pinterest categories, the YFCC100M includes all the Flickr
posts between 2004 and 2014. Thus, the topics of YFCC100M
are much wider than that of the InstaPIC-1.1M, thereby it is
more challenging to predict correct captions for YFCC100M.
Second, in the split by users for hashtag prediction, our ap-
proach (CSMN-*) significantly outperforms all the baselines.
Third, in the split by posts, the baseline (1NN-UsrIm) attains
the best results, mainly because of the severe variety of the
topics in the YFCC100M data, where nonparametric nearest-
neighbor would be better than other parametric models.
It is worth noting that the absolute metric values in
this work may be lower than those in the benchmark
datasets. Most benchmark datasets for image captioning
(e.g. Flickr30K [49] and MS COCO [50]) contain a limited
number of objects in the images, and multiple groundtruth
(GT) sentences per image. For example, the MS COCO
dataset contains 91 object categories, and five GT sentences
per image. On the other hand, our dataset is extremely
diverse and has only a single GT post and hashtag list per
test example. Such large diversity of our datasets may be the
main reason why the measured language metrics, which re-
quire word matches, are quite low. Similar tendency can be
also found in other highly diverse datasets with a single GT
sentence; one typical example may be LSMDC [52], a movie-
based video captioning dataset. As shown in its leader-
board1 , the best performing algorithm named (ELTanque)
shows 0.007 in BLEU-4, 0.055 in METEOR, 0.139 in ROUGE-
L, and 0.110 in CIDEr-D.
1. https://competitions.codalab.org/competitions/6121#results.
0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
11
5.4 Human Evaluation via Amazon Mechanical Turk
We perform two types of user studies to complement the
limitation of automatic langugage metrics in previous sec-
tion for evaluating the quality of generated sentences. We
include actual examples of AMT surveys in the supplemen-
tary material.
User Preferences. We perform AMT tests to observe
general users’ preferences between different algorithms for
the two post automation tasks. For each task, we randomly
sample 100 test examples. At test, we show a query image
and three randomly sampled complete posts of the query
user as a personalization clue, and two text descriptions
generated by our method and one baseline in a random
order. We ask turkers to choose the more relevant one
among the two for the query image and the query user. We
obtain answers from three different turkers for each query.
We select the variant (CSMN-P5) as a representative of our
method, because of its best quantitative performance. We
compare with three baselines by selecting the best method
in each group of 1NNs, image captioning, and language-
only methods: (1NN-UsrIm), (ShowTell), and (seq2seq).
Table 4 summarizes the results of AMT tests, which val-
idate that human annotators significantly prefer our results
to those of baselines. Among the baselines, (1NN-UsrIm) is
preferred the most, given that the performance gap with our
approach is the smallest. These results coincide with those
of quantitative evaluation in Table 2 and 3. Another reason
may be that the nearest-neighbor approach can always
retrieve grammatically correct text, which may be more
favorable to general turkers.
Sentence Quality. We perform another in-depth AMT
tests to evaluate several aspects of sentence generation,
using the similar setting proposed in [53]. We use the same
test set as in the user preference test. We ask turkers to rate
the quality of each output sentence from three aspects as
follows: (i)Plausibility: is the generated post plausible with
a given image? (ii)Grammaticality: is the post grammatically
correct? (iii)Relevance: does the post have a similar style to
the user’s previous posts? Each item is scored from 1 to 3,
where 1 means bad, 2 is acceptable, and 3 indicates good.
Table 5 summarizes the results of the AMT test, which
validate that human annotators assign significantly higher
scores to our results compared to those of the baselines. In-
terestingly, in grammaticality our model even earns higher
scores than the baseline, (1NN-UsrIm), which simply re-
trieves the nearest training sentence, which indicates the se-
quence generation capability of our model is also powerful
enough to be used as a language model.
5.5 Qualitative Results
Figure 5 illustrates selected examples of post generation. In
each set, we show a query image, GT, and generated text de-
scription by our method and baselines. In many of the Insta-
gram examples, GT comments are hard to predict correctly,
because they are extremely diverse, subjective, and private
conversations over a variety of topics. Nonetheless, most
of the predicted text descriptions are relevant to the query
images. Moreover, our CSMN model can appropriately use
normal words, emojis, and even mentions to other users
(anonymized by @username). Figure 6 shows examples of
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015
TABLE 4
AMT preference results for the two tasks between our methods and
three baselines for the InstaPIC-1.1M dataset. We show the
percentages of responses that turkers vote for our approach over
baselines.
Hashtag Prediction
vs. Baselines (1NN-UsrIm) (Showtell) (seq2seq)
(CSMN-P5) 67.0 (201/300) 88.0 (264/300) 81.3 (244/300)
Post Generation
(CSMN-P5) 73.0 (219/300) 78.0 (234/300) 81.3 (244/300)
TABLE 5
AMT test results for plausibility, grammaticality, and relevance of
generated posts by our method (CSMN-P5) and three baselines for the
InstaPIC-1.1M dataset. The score of each item ranges from 1 to 3. We
report averages and standard deviations (in parentheses).
Post Generation
Methods Plausibility Grammaticality
(1NN-UsrIm) 2.13 (0.55) 2.36 (0.54)
(Showtell) 2.26 (0.52) 2.34 (0.69)
(seq2seq) 1.98 (0.62) 2.10 (0.66)
(CSMN-P5) 2.36 (0.58) 2.43 (0.56)
hashtag prediction. We observe that our hashtag prediction
is robust even with a variety of topics, including profiles,
food, fashion, and interior design.
Figure 7 shows examples of how much hashtag and
post prediction varies according to different users for the
same query images. Although predicted results change in
terms of word usages and focused topics, all of the results
are relevant and meaningful to the query images. In the
fourth example of Figure 7, the first user focuses on family
while the third user focuses on the rugby game to generate
captions. Figure 8 illustrates the variation of text predictions
according to query images for the same user. We first select
a user whose most posts are about design, and then obtain
prediction by changing the query images. For design-related
query images, the CSMN correctly predicts relevant hash-
tags to the design topic (Figure 8(a)). For the query images
of substantially different topics, our CSMN is also resilient
to predict relevant hashtags to the images (Figure 8(b)).
5.6 Attention Visualization
Figure 9 shows three examples of user context memory
attention for hashtag prediction. In other words, they visual-
ize on which cells of the context memory is attended during
three time steps from t1 to t3 . Each example presents a query
image along with the GT and predicted hashtags by our
method, and the evolution of attention diagrams where the
words along the x-axis are the top-5 most attended words
in the memory cells at each time step, with darker colors
indicating more strongly attended memory cells. In the first
figure, our model generates #ipa #craftbeer #ballastpoint from
t1 to t3 , with attention on saison, spareribs, paleale, calgary,
ballastpoint in the memory, where all of the generated words
and strongly attentioned words both are related to the beer
topic. Similarly, in the second figure, our model produces
#actionfigures #mastersofuniverse #toys #hoardworld with at-
tention on batman, blogs, hoardax, actionfigures, allstarcomic-
smelbourne, dc, where, #mastersofuniverse is a comic name,
0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
12
#hoardworld is a famous comics blog name, and #dc is a
popular comics publisher. Third figure is another interesting
example in that our model generates #nz #kiwipics #nzmustdo
#newzealand with attention on bestvacations, thegreatoutdoors,
snorkeling, nzmustdo, newzealandfinds, nz, where #nz is an
abbreviation of New Zealand and #kiwipics is a frequently
used hashtag for pictures taken in New Zealand. Note that
output hashtags do not necessarily coincide with the most
attended words in the user context memory, although they
are likely to be correlated.
6 C ONCLUSIONS
We proposed the context sequence memory networks
(CSMN) as a first personalized image captioning approach.
We addressed two post automation tasks: hashtag predic-
tion and post generation. With quantitative evaluation and
Relevance AMT user studies on nearly collected InstaPIC-1.1M dataset
2.02 (0.58) and standard benchmark YFCC100M dataset, we showed
2.12 (0.56) that our CSMN approach outperformed other state-of-the-
1.88 (0.60) art captioning models.
2.25 (0.60)
There are several promising future directions that go
beyond the current paper. First, we can extend the model to
incorporate other meta-data such as location, gender or time
to generate more personalized sentences. Second, we can
extend the CSMN model for another interesting related task
such as post commenting that generates a thread of replies
for a given post. Third, since we dealt with only Instagram
and Flickr posts in this work, we can explore data in other
social networks such as Pinterest and Tumblr, which have
different post types, metadata, text and hashtag usages.
ACKNOWLEDGMENTS
This research is partially supported by Kakao, Kakao Brain
and Brain Research Program through the National Research
Foundation of Korea(NRF) funded by the Ministry of Sci-
ence, ICT & Future Planning (2017M3C7A1047860). Gunhee
Kim is the corresponding author. The code and dataset are
available at https://github.com/cesc-park/attend2u.
R EFERENCES
[1] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni,
D. Poland, D. Borth, and L.-J. Li, “YFCC100M: The New Data in
Multimedia Research,” in CACM, 2016.
[2] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach,
S. Venugopalan, K. Saenko, and T. Darrell, “Long-term Recurrent
Convolutional Networks for Visual Recognition and Description,”
in CVPR, 2015.
[3] H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollar,
J. Gao, X. He, M. Mitchell, J. Platt, L. Zitnick, and G. Zweig, “From
Captions to Visual Concepts and Back,” in CVPR, 2015.
[4] A. Karpathy and L. Fei-Fei, “Deep Visual-Semantic Alignments for
Generating Image Descriptions,” in CVPR, 2015.
[5] R. Kiros, R. Salakhutdinov, and R. Zemel, “Multimodal Neural
Language Models,” in ICML, 2014.
[6] V. Ordonez, G. Kulkarni, and T. L. Berg, “Im2Text: Describing
Images Using 1 Million Captioned Photographs,” in NIPS, 2011.
[7] C. C. Park and G. Kim, “Expressing an Image Stream with a
Sequence of Natural Sentences,” in NIPS, 2015.
[8] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and
Tell: Lessons Learned from the 2015 MSCOCO Image Captioning
Challenge,” in IEEE TPMAI, 2016.
[9] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov,
R. Zemel, and Y. Bengio, “Show, Attend and Tell: Neural Image
Caption Generation with Visual Attention,” in ICML, 2015.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015 13

User Context Memory User Context Memory User Context Memory

ipa actionfigures nzmustdo
ascmelbourne
hoardax nzgreatwalks
calgary
allstarcomicsmelbourne nz
b
neca tramping
nativetonguesyyc vhs peoplewhodofunstuff
blogs bestvacations
comicbooks stokedforsaturday
ebay discoverqueenstown
ballastpoint boxes liveclimbrepeat
spareribs dc nzac
xe0 batman newzealandfinds
freshness
saison snorkeling
thegreatoutdoors
paleale
ferry
rockclimbing
waterfalls
(GT) #ipa #fourwindsbrewco (GT) #timburton #videogame (GT) #omarama #liveoutdoors waves
familydinner
#craftbeer #fourwindsbrewing #actionfigures #batman #thegreatoutdoors #nzmustdo
(Ours) #michaelkeaton #neca #toys #kiwipics #claycliffs
𝑡" : #ipa #hoardworld #keaton #newzealand
𝑡# : #craftbeer potatoes (Ours) (Ours)
𝑡$ : #ballastpoint pork 𝑡" : #actionfigures 𝑡" : #nz
𝑡# : #mastersoftheuniverse 𝑡# : #kiwipics
𝑡$ : #toys 𝑡$ : #nzmustdo
𝑡" 𝑡# 𝑡$ 𝑡% : #hoardworld 𝑡" 𝑡# 𝑡$ 𝑡% 𝑡% : #newzealand 𝑡" 𝑡# 𝑡$ 𝑡%

Fig. 9. Three examples of user context memory attention for hashtag prediction. We show the query image along with the groundtruth and predicted
hashtags by our method. On the right, we present the evolution of attention diagrams for three time steps from t1 to t3 , where the words along
the x-axis are the top-5 most attended words in the memory cells at each time step, with darker colors indicating more strongly attended memory
cells. Note that output hashtags do not necessarily coincide with the most attended words in the user context memory, although they are likely to
be correlated.

[10] Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image Captioning
with Semantic Attention,” in CVPR, 2016.
[11] C. Fan and D. Crandall, “DeepDiary: Automatic Caption Genera-
tion for Lifelogging Image Streams,” in ECCV EPIC, 2016.
[12] J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller,
R. Miller, A. Tatarowicz, B. White, S. White et al., “VizWiz: Nearly
Real-Time Answers to Visual Questions,” in UIST, 2010.
[13] K. Ramnath, S. Baker, L. Vanderwende, M. El-Saban, S. N. Sinha,
A. Kannan, N. Hassan, M. Galley, Y. Yang, and D. Ramanan, “Au-
tocaption: Automatic Caption Generation for Personal Photos,” in
WACV, 2014.
[14] S. Zhang, E. Dinan, J. Urbanek, A. Szlam, D. Kiela, and J. Weston,
“Personalizing Dialogue Agents: I Have a Dog, Do You Have Pets
Too?” in arXiv:1801.07243, 2018.
[15] A. Graves, G. Wayne, and I. Danihelka, “Neural Turing Machines,”
in arXiv:1410.5401, 2014.
[16] S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus, “End-to-End
Memory Networks,” in NIPS, 2015.
[17] J. Weston, S. Chopra, and A. Bordes, “Memory Networks,” in
ICLR, 2015.
[18] I. Sutskever, O. Vinyals, and Q. Le, “Sequence to Sequence Learn-
ing with Neural Networks,” in NIPS, 2014.
[19] R. J. Williams and D. Zipser, “A Learning Algorithm for Continu-
ally Running Fully Recurrent Neural Networks,” in NIPS, 1989.
[20] O. Vinyals, Ł. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton,
“Grammar as a Foreign Language,” in NIPS, 2015.
[21] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a Method
for Automatic Evaluation of Machine Translation,” in ACL, 2002.
[22] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-
based Image Description Evaluation,” in CVPR, 2015.
[23] S. B. A. Lavie, “METEOR: An Automatic Metric for MT Evaluation
with Improved Correlation with Human Judgments,” in ACL,
2005.
[24] C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Sum-
maries,” in WAS, 2004.
[25] C. C. Park, B. Kim, and G. Kim, “Attend to You: Personalizd Image
Captioning with Context Sequence Memory Networks,” in CVPR,
2017.
[26] Z. Ren, X. Wang, N. Zhang, X. Lv, and L.-J. Li, “Deep Re-
inforcement Learning-based Image Captioning with Embedding
Reward,” in CVPR, 2017.
[27] S. Venugopalan, L. A. Hendricks, and M. Rohrbach, “Captioning
Images with Diverse Objects,” in CVPR, 2017.
[28] J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing When to Look:
Adaptive Attention via a Visual Sentinel for Image Captioning,”
in CVPR, 2017.
[29] E. Denton, J. Weston, M. Paluri, L. Bourdev, and R. Fergus, “User
Conditional Hashtag Prediction for Images,” in KDD, 2015.
[30] P.-L. Hsieh, C. Ma, J. Yu, and H. Li, “Unconstrained Realtime Facial
Performance Capture,” in CVPR, 2015.
[31] T. Almaev, B. Martinez, and M. Valstar, “Learning to Transfer:
Transferring Latent Task Structures and Its Application to Person-
specific Facial Action Unit Detection,” in ICCV, 2015.
[32] A. Yao, L. Van Gool, and P. Kohli, “Gesture Recognition Portfolios
for Personalization,” in CVPR, 2014.
[33] W. Kienzle and K. Chellapilla, “Personalized Handwriting Recog-
nition via Biased Regularization,” in ICML, 2006.
[34] O. Polozov, E. ORourke, A. M. Smith, L. Zettlemoyer, S. Gulwani,
and Z. Popovic, “Personalized Mathematical Word Problem Gen-
eration,” in IJCAI, 2015.
[35] S. Mirkin, S. Nowson, C. Brun, and J. Perez, “Motivating
Personality-aware Machine Translation,” in EMNLP, 2015.
[36] A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka,
A. Grabska-Barwińska, S. G. Colmenarejo, E. Grefenstette, T. Ra-
malho, J. Agapiou et al., “Hybrid Computing Using a Neural
Network with Dynamic External Memory,” in Nature, 2016.
[37] A. Kumar, O. Irsoy, J. Su, J. Bradbury, R. English, B. Pierce,
P. Ondruska, I. Gulrajani, and R. Socher, “Ask Me Anything:
Dynamic Memory Networks for Natural Language Processing,”
in ICML, 2016.
[38] A. Miller, A. Fisch, J. Dodge, A.-H. Karimi, A. Bordes, and
J. Weston, “Key-Value Memory Networks for Directly Reading
Documents,” in EMNLP, 2016.
[39] Y. Kim, “Convolutional Neural Networks for Sentence Classifica-
tion,” in EMNLP, 2014.
[40] S. Lai, L. Xu, K. Liu, and J. Zhao, “Recurrent Convolutional Neural
Networks for Text Classification,” in AAAI, 2015.
[41] Y. Kim, Y. Jernite, D. Sontag, and A. M. Rush, “Character-Aware
Neural Language Models,” in AAAI, 2016.
[42] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin,
“Convolutional Sequence to Sequence Learning,” in ICML, 2017.
[43] J. Weston, S. Chopra, and K. Adams, “Tagspace: Semantic Embed-
dings from Hashtags,” in EMNLP, 2014.
[44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for
Image Recognition,” in CVPR, 2016.
[45] V. Nair and G. E. Hinton, “Rectified Linear Units Improve Re-
stricted Boltzmann Machines,” in ICML, 2010.
[46] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language
Modeling with Gated Convolutional Networks,” in ICML, 2017.
[47] D. P. Kingma and J. L. Ba, “ADAM: A Method For Stochastic
Optimization,” in ICLR, 2015.
[48] W. Zaremba and I. Sutskever, “Learning to Execute,” in
arXiv:1410.4615, 2014.

0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2018.2824816, IEEE
Transactions on Pattern Analysis and Machine Intelligence
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO 8, AUGUST 2015 14

[49] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From Image

Descriptions to Visual Denotations: New Similarity Metrics for
Semantic Inference over Event Descriptions,” in TACL, 2014.
[50] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Dollar, and C. L. Zitnick, “Microsoft COCO: Common Objects
in Context,” in ECCV, 2014.
[51] T. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal,
J. Devlin, R. B. Girshick, X. He, P. Kohli, D. Batra, C. L. Zitnick,
D. Parikh, L. Vanderwende, M. Galley, and M. Mitchell, “Visual
Storytelling,” in NAACL HLT, 2016.
[52] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal,
H. Larochelle, A. Courville, and B. Schiele, “Movie Description,”
in IJCV, 2017.
[53] K. Guu, T. B. Hashimoto, Y. Oren, and P. Liang, “Generating
Sentences by Editing Prototypes,” in arXiv:1709.08878, 2017.

Cesc Chunseong Park is currently working as

a researcher at Lunit Inc. He received the M.S.
degree from Computer Science and Engineering
at Seoul National University in 2017. He is a
recipient of the 2016 Naver MS Fellowship and
NVIDIA Deep Learning Contest Awards (Korea).
His research interests span the intersection of
deep learning, computer vision and natural lan-
guage. In particular, he is interested in defining
and solving new problems and data rather than
standard tasks and data.

Byeongchang Kim is an M.S. student in Com-

puter Science and Engineering at Seoul Na-
tional University. Before that, he received the
B.S. degree in Biosystems Engineeraing & Com-
puter Science and Engineering at Seoul National
University. His research interests include deep
learning, computer vision and natural language
understanding.

Gunhee Kim received the BS degree in me-

chanical engineering from the Korea Advanced
Institute of Science and Technology (KAIST),
the MS degree from Robotics Institute, CMU,
and the PhD degree from the Computer Sci-
ence Department, Carnegie Mellon University,
in 2013. He has been an assistant professor in
computer science and engineering of Seoul Na-
tional University since 2015. Prior to that, he was
a postdoctoral researcher at Disney Research
Pittsburgh for one and a half years. His research
interests include computer vision, machine Learning, data mining, and
robotics. He received the 2014 ACM SIGKDD doctoral dissertation
award, and 2015 Naver New faculty award.

0162-8828 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.