Reference-based Manga Colorization by Graph Correspondence
Using Quadratic Programming
Kazuhiro Sato Yusuke Matsui Toshihiko Yamasaki
The University of Tokyo
Abstract
Manga (Japanese comics) are popular all over the world. However,
most existing manga are monochrome. If such monochrome manga
can be colorized, readers can enjoy the richer representations. In
this paper, we propose a semiautomatic colorization method for
manga. Given a previously colored reference manga image and tar-
get monochrome manga images, we propagate the colors of the ref-
erence manga to the target manga by representing images as graphs
and matching graphs. The proposed method enables coloring of
manga images without time-consuming manual colorization. We
show results in which the colors of characters were correctly trans-
ferred to target characters, even those with complex structures.
CR Categories: I.4.3 [Image Processing and Computer Vision]:
Enhancement—Grayscale Manipulation I.4.9 [Image Processing
and Computer Vision]: Application;
Keywords: Manga, Colorization, Graph matching
1 Introduction
Manga (Japanese comics) are popular and enjoyed all over the
world. Manga have been created by pen and ink on white pa-
per, and most existing manga are monochrome. Recently, colored
manga have become more popular, and colorizing monochrome
manga has attracted the attention of manga publishers and authors.
If previously published monochrome manga are colorized and re-
distributed, they gain readers’ attention and readers can enjoy the
richer representations of manga.
However, colorization of manga is a tedious and time-consuming
task that must be performed manually using image editing software
such as Photoshop or by scribble-based manual colorization meth-
ods [Levin et al. 2004; Sýkora et al. 2009]. Although automatic
colorization methods for natural images exist [Gupta et al. 2012;
Irony et al. 2005; Welsh et al. 2002], it is hard to apply them to
manga because their work is based on the assumption that neigh-
boring pixels that have similar intensity have similar colors, which
is not true for manga. The nature of manga images is very differ-
ent from that of natural images because manga consist only of line
drawings and textural patterns. Qu and colleagues [Qu et al. 2006]
proposed a colorization method for manga that manually colorizes
plain and textural pattern regions in a unified manner. However,
in their method each area of a manga image still requires an initial
operation of manual color assignment by user scribbles, so their
method is still time-consuming when many manga images need to
be colorized.
∗ e-mail:{sato, matsui, yamasaki, aizawa}@hal.t.u-tokyo.ac.jp
Kiyoharu Aizawa∗
In this paper, we propose a semiautomatic colorization method tai-
lored for manga. Given a reference colored manga character and
target monochrome characters, our method automatically propa-
gates the colors of the reference character to the target characters1 .
Our proposal dramatically reduces the time required for colorizing
manga characters. Our method has three parts. First, we segment
regions of both reference and target images, and represent relations
between regions by a graph structure. This graph represents the rel-
ative positions of parts of a manga character. Second, nodes of ref-
erence and target graphs are matched by constrained 0-1 quadratic
programming. We formulate the matching so as to maintain loose
relative positions of the parts of the character. Finally, because 0-1
quadratic programming is NP-hard, we solve the matching approx-
imately by relaxing the variables into the continuous space.
2 Proposed Approach
Figure 1 shows an overview of the system. The reference is colored
beforehand. First, in preprocessing, the reference and target images
are segmented. Next, both images are represented as graphs. Fi-
nally, the graphs are matched to find correspondences and colors
are propagated by quadratic programming.
2.1 Segmentation
In preprocessing, we segment the reference and target images by
computing superpixel-based filling and segmentation [Shi and Ma-
lik 1997]. Because a manga image usually consists of flat regions,
this tends to work well. For example, a face of a character in Fig-
ure 1 is a single, flesh-colored, flat region. Users can modify results
if they are not satisfied.
2.2 Finding Corresponding Regions by Graph Match-
ing
In manga, the shape and the position of a region are often deformed.
Therefore, when we try to match a region in the target image to a
region in the reference image, it is difficult to find the best matching
simply by comparing the regions one by one. Thus, we aim to find
the overall correspondence using relative positions between neigh-
boring regions and keeping the two images as similar as possible.
In the proposed method, regions in the colored reference image and
a monochrome target image are represented as nodes in a graph
structure. We call these the reference graph and target graph, re-
spectively. Then, colorization is performed by formulating the
matching of the two graphs as a constrained 0-1 quadratic program-
ming problem.
We denote the reference graph as Gr = (Vr , Er ) and the target
graph as Gt = (Vt , Et ), where V represents a set of nodes and E
represents a set of edges. Examples are shown in Figure 1. A so-
lution can be defined as a permutation matrix X = (xij )Nr ×Nt ∈
{0, 1}Nr ×Nt , where Nr and Nt denote the numbers of nodes in
Gr and Gt , respectively.
1 Whilecharacters are considered here, our method can be extended to
other objects.
 Graph matching
Segmentation

Reference image (a) (b) Reference (c) (d) (e)

graph 𝐺𝑟

Colorized Colorized
target target image
graph 𝐺𝑡
Target image Target
graph 𝐺𝑡

Figure 1: System overview. (a) The colored reference image and a target image are segmented into small regions. This can be done by simple
superpixel-based filling. (b) A graph that represents the image is constructed in which each region is represented by a node, for both reference
and target images. (c) The reference and target graphs are matched. (d) Each node of the target graph is colored. (e) By using the colored
target graph, the target image is colored.

When the ith region in the reference image (ith node from Vr ) cor-
responds to the jth region in the target image (jth node from Vt ),
xij is set to one, and zero otherwise. For one-to-one correspon-
dence, each row and column of X must satisfy the conditions:
X X
xij = 1, xij = 1. (1)
j i

If the two graphs have different numbers of nodes, dummy nodes

are added so that X is a square matrix (discussed further below).
To match Gr and Gt , we define a cost function Q as:
Figure 2: Relation between graph nodes
Q(i1 , j1 , i2 , j2 ) = Dlength · Dangle · Darea , (2)
 
|vi1 i2 | − |vj1 j2 |
 
Dlength = , (3) If the two graphs have different numbers of nodes, we adjust the
max(vi1 i2 , vj1 j2 ) size of matrix Q by inserting dummy nodes so that Q will be a
square matrix of dimension N 2 , where N = max(Nr , Nt ). It
 
1 vi i · vj1 j2
Dangle = 1− 1 2 , (4) must be noted that cost Q(i1 , j1 , i2 , j2 ) becomes effective if and
2 |vi1 i2 ||vj1 j2 |
only if correspondences xi1 j1 = 1 and xi1 j1 = 1 both exist. If
|ai1 aj2 − ai2 aj1 | xi1 j1 · xi2 j2 = 0, the cost of this pair is not counted (see Eq. 6).
Darea = . (5)
max(ai1 aj2 , ai2 aj1 )
An example is shown in Figure 2. The three measures, 2.3 Relaxation and Convexification
Dlength , Dangle , and Darea , evaluate the difference between two
pairs (i1 , i2 ) and (j1 , j2 ). vi1 i2 is the vector between the centroids Equation 7 describes constrained 0-1 quadratic programming,
of the two regions i1 and i2 , and ai1 is the area of region i1 . If the which is NP-hard and cannot be solved in polynomial time [Ham-
two pairs have similar relative positions and areas, the cost becomes mer and Rubin 1970]. Therefore, we convexify the objective func-
2
small. If either pair includes a dummy node, the cost is set as the tion xt Qx in Eq. 7 and relax the binary variables x ∈ {0, 1}N
2
maximum value 1 (Q(i1 , j1 , i2 , j2 ) = 1). This is to avoid incorrect of Eq. 7 into continuous ones x ∈ [0, 1]N . The variable
matching with dummy nodes. 2
x ∈ {0, 1}N satisfies x2ij = xij . Thus, Eq. 7 and the follow-
Based on this idea, the solution of the matching is defined as the ing modified Eq. 8 are equivalent:
vector x that minimizes the total sum (Eq. 6) under the one-to-one
N
constraints (Eq. 7): X
min xt (Q − λmin I)x + λmin xij
P t
i1 ,j1 ,i2 ,j2 Q(i1 , j1 , i2 , j2 ) · xi1 j1 · xi2 j2 = x Qx, (6) i,j=1
2 2
min xt Qx s. t. Cx = 1, x ∈ {0, 1}N . (7) s. t. Cx = 1, x ∈ {0, 1}N . (8)

Here, x is a concatenation of column-wise vectors of matrix X, Q
is a matrix containing Q(i1 , j1 , i2 , j2 ) as its elements, and Cx =
1, where 1 is a column vector of all ones and C is a matrix that
imposes one-to-one correspondences between the two sets of nodes
(equivalent to Eq. 1).
Here, λmin is the smallest eigenvalue of matrix Q and I is an iden-
tity matrix. Thus, the expression Q − λmin I is positive semidefi-
nite, and the objective function becomes convex. Any number with
value −λmin or greater can be used for the diagonal matrix to con-
vexify the objective function; however, it can be shown that the
error caused by the relaxation is minimized when −λmin is used
[Hammer and Rubin 1970].
By relaxing the variable and replacing the second term using the
vector notation λmin · 1 = λmin , Eq. 8 above becomes:
min xt (Q − λmin I)x + λtmin x
2 Reference image Target image Colorization result
s. t. Cx = 1, x ∈ [0, 1]N . (9)
(a) Even though the shapes of the hands change greatly, they are correctly
Equation 9 can easily be solved using the interior-point-convex al- matched.
gorithm.
Once the solution is obtained in the continuous domain, we must
convert it back to the original binary domain. Before that, we again
2
represent the continuous solution xc ∈ [0, 1]N in a matrix form
N ×N
Xc ∈ [0, 1] . To obtain the binary solution Xb ∈ {0, 1}N ×N
from Xc , we choose one element from each row and column of Xc
without duplication to satisfy Eq. 1.
Reference image Target image Colorization result
3 Colorization Results
(b) The character’s elbows do not appear in the target image, but the arm
We tried to color more than 30 images. In this paper, only some parts are correctly colored.
sample images are shown because of space limitations. We show
colorization results in Figures 3(a) to 5(c). In all cases, the in- Figure 3: Successful results (1)
put reference and target images of manga characters were cropped
from manga titles (or reproductions) and all reference images were
colored. In preprocessing, we segmented the target characters into
small parts automatically as discussed in Section 2.1. If the result We also show an application of our proposed method to key frames
was not good, we modified the result manually. Then, we applied of an animation. In key frames, characters have small posture
the proposed colorization method to the target images. This step changes between successive images in one scene, and the proposed
was automatic and was completed very quickly. method can work well. An example is shown in Figure 6.

We show successful results in Figure 3(a)–4(b). In Figure 3(a),

although the shapes of the mouth and hands are changed by the
posture change of the character, they were correctly colored. Fig-
ure 3(b) and 4(a) also show successful cases. The visible parts of the
shirts of the reference and target characters in Figure 3(b) have dif-
ferent shapes, but they were correctly colored. Similarly, the arms
of the reference and target in Figure 4(a) are correctly colored, even
though their positions are quite different. In Figure 4(b), although
part of the target character’s face was not colored, almost all body
parts are colored correctly. In particular, the right arm in the refer-
ence image had complex shading patterns, but the target image did
not have this structure. It is hard to find exact correspondences in
such cases, but the proposed method automatically assigned correct
correspondences so as to keep the total error to a minimum.
Examples of failures are shown in Figure 5(a)–5(c). In Figure 5(a),
some regions in the reference image disappear or are split in the
target and this has led to incorrect correspondences in part of the
straw hat. Figure 5(b) also shows a partial failure. The method
tends to fail when the image contains repetitive patterns such as the
hair band in Figure 5(b). In Figure 5(c), the face and forehead of
the character were divided in the reference but merged in the target,
which caused the failure.
The results show that the proposed method can color manga images
well even when postures are changed. The method does not work
well when a region is split into multiple small regions, or when
multiple regions are merged into a single one. However, even if
the result is not perfect, it is a good initial guess and requires only
small manual modifications to complete a colorization task. The
accuracy is not discussed because it depends on the similarity be-
tween the reference and target images, the superpixel process, and
other aspects. Typical processing time is a few seconds to tens of
seconds. For example, Figure 3(a) with Nt = 17 took a few sec-
onds, while Figure 6 with Nt = 36 took about 20 seconds in a
MATLAB implementation.
4 Conclusions
We have proposed a novel method to color manga images from col-
ored reference images. In the proposed method, we represent re-
gions as nodes of a graph structure and we color the monochrome
target image by matching the graphs of the reference and target im-
ages. The proposed method colors manga characters properly, even
if they have some posture changes.
Acknowledgements: We thank Fujiko-Pro and CELSYS for the
use of their images, Figure 4(a) and Figure 6, respectively. This
work was partially supported by the Strategic Information and
Communications R&D Promotion Programme (SCOPE) and JSPS.
References
G UPTA , R. K., C HIA , A. Y.-S., R AJAN , D., N G , E. S., AND
Z HIYONG , H. 2012. Image colorization using similar images.
In Proceedings of ACM Multimedia, 369–378.
H AMMER , P., AND RUBIN , A. 1970. Some remarks on quadratic
programming with 0-1 variables. RAIRO - Operations Research
- Recherche Opérationnelle 4, 67–79.
I RONY, R., C OHEN -O R , D., AND L ISCHINSKI , D. 2005. Col-
orization by example. In Eurographics Conference on Rendering
Techniques, 201–210.
L EVIN , A., L ISCHINSKI , D., AND W EISS , Y. 2004. Colorization
using optimization. ACM Transactions on Graphics 23, 3, 689–
694.
Q U , Y., W ONG , T.-T., AND H ENG , P.-A. 2006. Manga coloriza-
tion. ACM Transactions on Graphics 25, 3, 1214–1220.
 Target image 1 Target image 2 Target image 3
(t=1) (t=2) (t=3)

Once a wrong correspondence occurs,

it remains in the following frames.

Reference image Colorized target image 1 Colorized target image 2 Colorized target image 3
(t=0) (t=1) (t=2) (t=3)

Figure 6: Colorization of animation images. Target image 1 (t=1) is colorized using the reference image (t=0), target image 2 (t=2) is
colorized using colorized target image 1 (t=1), and target image 3 (t=3) is colorized using colorized target image 2 (t=2).

Disappears wrong
in the target
image

Splits in the
target image

Reference image Target image Colorization result

(a) Part of the straw hat fails because there is a balloon in the target image
(a) The positions of the arms are changed but they are properly colored. and the straw hat is partly covered and divided. With such splits or merges,
the proposed method tends to fail.

This regions has no

corresponding region,
so it is not colored.

+ =
Reference image Target image Colorization result

(b) The hair band is not correctly colored because the number of patterns in
Reference image Target image Colorization result it is different. Note that other parts, like the mouth, are properly colored.
(b) Although part of the hair is not colored, almost all of the body is correctly
colored. Merge in the
target image Failure
Figure 4: Successful results (2)

S HI , J., AND M ALIK , J. 1997. Normalized cuts and image seg- + =

mentation. In Proceedings of IEEE Computer Vision and Pattern
Recognition, 731–737.
S ÝKORA , D., D INGLIANA , J., AND C OLLINS , S. 2009. Lazy- Reference image Target image Colorization result
brush: Flexible painting tool for hand-drawn cartoons. Computer
(c) Because the face and forehead of the character are divided in the reference
Graphics Forum 28, 2, 599–608.
but merged in the target, the colorization fails. FUJIO
c PRO

W ELSH , T., A SHIKHMIN , M., AND M UELLER , K. 2002. Trans-

ferring color to greyscale images. ACM Transactions on Graph- Figure 5: Partial failures
ics 21, 3, 277–280.