New GenerationComputing,23(2005)181-191

Ohmsha, Ltd. and Springer

ERATION
iMPUTING
9 Ltd. 2005

Music Composition by Interaction between Human and

Computer
Muneyuki U N E H A R A and Takehisa ONISAWA
Onisawa Laboratory,
Graduate School of Systems and Information Engineering,
University of Tsukuba, 1-1-1 Tennodai, Tsukuba, lbaraki 305-8573, Japan
{une, onisawa}~fhuman, e s y s . t s u k u b a , ac. j p

Received 26 April 2004

Revised manuscript received 10 September 2004

Abstract This paper aims at constructing a music composition sys-

tem that composes music by the interaction between human and a computer.
Even users without special musical knowledge can compose 16-bar musical
works with one melody part and some backing parts using this system. The
interactive Genetic Algorithm is introduced to music composition so that
users' feeling toward music is reflected in the composed music. One chro-
mosome corresponds to 4-bar musical work information. Users participate in
music composition by evaluating composed works after GA operators such as
crossover, mutation, virus infection are applied to chromosomes based on the
evaluation results. From the experimental results, it is found that the users'
evaluation values become high over the progress of generations. That is, the
system can compose 16-bar musical works reflecting users' feeling.

Keywords: Music Composition, Interactive Genetic Algorithm, Feeling, Sub-

jectivity, Human-Centered System, Interaction.

w Introduction
It is well known that human feelings are influenced by music. There have
been many studies in various fields about the relationship between music and
human feelings, 4) e.g., studies about music therapy and mood generation in living
environments using background music. There also has been much research on
automatic music composition using computers. 2'~ Music composition is usually
a complex and hard task for novices who don't have musical knowledge or skill,
since the composer's expertise plays an important role in music composition.
Some automatic music composition systems for beginning composers have been
182 M. U n e h a r a a n d T. O n i s a w a

proposed. 1~) However, a musical work composed by an automatic composition

technique does not give the same impression to all listeners because of human
subjectivity toward music. 1H3~ Therefore, automatic music composition systems,
by which inexpert users can compose original musical works reflecting their own
preferences, need to be developed.
This study aims at the construction of a music composition system con-
sidering the following two points. The first is that the composition system helps
musically unskilled users compose their own music. The second is that the sys-
tem reflects users' own subjectivity using their direct evaluation of musical works.
To this end the Interactive Genetic Algorithm (abbreviated as Interactive GA)
is applied to music composition. The Interactive GA with human subjective
evaluation instead of fitness functions is often applied to the fields such as arts
or industrial design, io~
This paper describes an interactive music composition system that com-
poses 16-bar musical works reflecting users' feeling. The present system is an
extended version of a previous system composing only 4-bar musical works. 14'~5>
Users compose a favorite 16-bar musical work with one main melody part and
several accompaniment (backing) parts, using the present system. Music com-
position with Interactive GA is described in Section 2. The structure of the
musical work and how to generate primal musical works are described in Section
3. The detail of the system structure is described in Section 4, including an
outline of music composition, users' evaluations, and GA operators. Subjects'
experiments to confirm the validity of the system are summarized in Section 5.

w Music Composition with Interactive Genetic Algorithm

Recently, many applications of GA or Interactive GA to the field of music
composition have been created. 17) Nelson tries to find and compose structured
musical works using GA. 7) This system uses fitness functions to compose musical
works. Biles constructed an interactive improvisation system named "GenJam",
which is an agent that plays jazz music with a human jazz player. 1) Unemi con-
structed an interactive music composition support system named "SBEAT", 1~)
which generates several kinds of 1-bar musical works and breeds them based on
users' evaluations. This technique is called simulated breeding. If users want
to compose longer musical works with more than 2 bars using the system, they
must connect some bars for themselves. Tokui constructed a system that gen-
erates a rhythm section (i.e. drum set).12) In this system, the neural networks
obtained by some learning method play the role of fitness functions to evalu-
ate the works. Johanson also tried to compose music automatically using GP
(Genetic Programming) with fitness functions. 3)
The present study constructs a music composition system as a human-
centered system, in which human subjectivity or KANSEI plays a central role. 8)
Interaction between human and a computer is the central part of the present
system, and features of the system are that it has simple and understandable
operations among users. This human centered approach is the point of view
disregarded in the previously developed applications that mainly aim at users
Music Composition by Interaction between Human and Computer 183

having musical knowledge and experience. Even inexpert users can compose
their own favorite works using the present system.

w Structure o f C o m p o s e d Musical Works

In this section, the structure of composed musical works with the present
system is described. T h e system automatically generates primal musical works
with 4 bars. Users finally compose a favorite 16-bar musical work with one main
melody part and several accompaniment (backing) parts using this system.

3.1 Length and Part Structure

In order to compose popular music, the length of a musical work is fixed
to be 16 bars in this paper. A musical work with 16 bars is usually called "two-
sentence" music, which is common among children's songs and folk songs. 2~
Sixteen-bar musical works give listeners more varied impressions than 4-bar
works. The musical part structure is shown in Table 1, which is one of com-
monly used patterns among today's musical works. The melody p a r t has a main
melody line, which gives listeners various impressions. Backing parts 1-3 are
accompaniments playing fixed patterns, and the backing part 4 is the rhythm
part played by drums. In this paper the p a t t e r n of backing parts is assumed to
be fixed as "Rock and Roll N o . l ' , which is chosen from a general backing parts'
pattern book. 18/ In this paper the pitch of notes is assumed to be chosen from
a C-major scale. 6> T h e pitches in the C-major scale are shown as c, d, e, f, g, a,
or b, corresponding to do, re, mi, fa, sol, ra, or si respectively, and the octaves
are shown as 1, 2, 3, 4, or 5. The range of the note pitch in each p a r t is fixed as
shown in Table 1.
Table1 Musical Part Structure
Part Name Role Pitch Range
Melody Main f4-g5
Backing Part 1 Guitar f3-g4
Backing Part 2 Keyboard f3-g4
Backing Part 3 Bass f2-g3
Backing Part 4 Drums

3.2 Generation of Primal 4-Bar Musical Works

Listeners usually feel musical works with 4 bars as either an ending or a
turning point of music. Therefore, the system automatically generates primal
musical works with 4 bars. Music theory 6/ is applied to the generation of the
works in order to have rich expressions and give various kinds of impressions to
listeners. This system assigns a chord to a bar based on the theory of harmonic
progression 5'6) for the generation of a work in which all musical parts harmonize
with each other.IS) In order to generate structured music that sounds musically
natural to listeners, a melody line is generated combining several blocks that
consist of sets of notes (i.e., 1 ~,, 4 kinds of notes). 1~ The pitch of a note in the
melody is determined by considering assigned chord in a bar and the theoretical
184 M. Unehara and T. Onisawa

facts mentioned in the music theory9~)

3.3 Coding of 4-Bar Musical Works

Figure 1 shows the structure of a chromosome used in this paper and an
example of a 4-bar musical work. Three structural parts are considered in one
chromosome, where the first part expresses the tone color of each part and the
chord name, the second part expresses the pitch and the length of notes in
the melody part, and the third part expresses the notes in the backing 15arts.
The melody part plays an important role to give listeners a feeling of music.
Therefore, in this paper the GA operators are applied to the melody part of the
chromosome. However, some backing parts are also varied. In the third Part. of
the backing parts, although each note length is fixed according to the "Rock and
Roll No.l" pattern, its pitch is determined according to the chord name in the
first part colored green in Fig. 1. The chromosome length of each part is varied
according to that of the melody part in the codification.

One chromosomecorrespondsto a 4-barmusicalwork . ..... ~tch -Bioc'l~""""-..

',, ~ 2, ~-

Tone Color of Each Part ] [ Pitch / Length of Each ! '~.......... 1 Pitch / Length of Each
]/C~:me ~Melody Part i "/9 [ Note i n % B ~ r t s

~.o_,CM~, J...........................I Chromosome

:F~7 ~ .... ~TZll

Go~ta~~ ::::::::::::::::::::::::::::::::::::::::::::::::::::::

i l ~ ~ ~ ~ ~ ~ ~]Fourbar

Fig. 1 Chromosomeand 4-Bar Musical Work

w System Structure

4.1 Outline of Music Composition

Figure 2 shows the interaction between users and the present system in
the composition of 16-bar musical works. The outline of music composition is as
follows. 1) Users have a feeling of music they want to compose, e.g., "Cheerful
Music", and start up the system. 2) The system generates 200 primal chromo-
somes automatically using the database based on music theory. Twelve chro-
mosomes, i.e., twelve 4-bar musical works are chosen at random and presented
to users through the user interface. 3) Users listen to them and give subjective
Music Composition by Interaction between Human and Computer 185

Fig. 2 Outline of Music Composition

evaluations to them. If users find favorite 4-bar works, they copy t h e m as "Users
Favorites". Four favorite works are connected and composed as a 16-bar musical
work. 4) The system applies GA operators to the chromosomes based on users'
subjective evaluations in order to generate new works reflecting users' feeling of
music. Then twelve chromosomes are presented to users again. Procedures 3)
and 4) are repeated until users compose a satisfactory 16-bar musical work.

4.2 Generation and Presentation of Primal Musical Works

In order to compose 16-bar musical works that have melody development
with rich representation, this system prepares four pools of chromosomes, and
evolves chromosomes in the pools independently. The 1st ~ 4th pools are pre-
pared for 4-bar musical works of the 1st ~,, 4th phrases. In each pool, 50 chromo-
somes, i.e., the works with 1st ~ 4th bars, 5th ~ 8th bars, 9th ~ 12th bars, and
13th ~ 16th bars in the 16-bars work, are generated in order to keep diversity
of chromosomes characteristics. Three chromosomes are chosen at r a n d o m from
each pool. Therefore, twelve chromosomes are presented to users.

4.3 User Interface and Subjective Evaluation

The user interface of the present system is shown in Fig. 3. Twelve gen-
erated works are presented in the tab window. Each page in the t a b window
presents three kinds of 4-bar musical works. Each note in the melody line is
shown by the rectangle of which vertical position represents the pitch and hor-
izontal length represents the length. Users can get information on the melody
line only seeing these rectangles, not listening to the work and without musi-
cal knowledge of reading music scores. For example, "this work has m a n y long
notes", "this work is similar to the work I saw before", and so on. After the
second generation, GA operated parts of melody lines are colored so that users
can see which parts are changed. Presented colors are green, blue, or yellow,
which mean crossovered parts, mutated parts, or virus-infected parts, respec-
tively. Users listen to the works and evaluate t h e m through the interface.
186 M. Unehara and T. Onisawa

. . . . ! displaying melody hne

........ [~ c:~ r - _ . - - - - ) o ~ c~ c : : ~ c = ~ ~ r~ I
STEP t :

.fS=-.'~ I C?J

if ~ u find Very Good M~sic, m ~ ~ l m m

~oo can p~ese~w it in the
i~W~r pert of the Interface,
aS " U~er's Favorite.* Plemm
d r e l and drop tim ~ b~tor !
of the mimic.
,= O

I! ~ U (,In f i l l ~ o d o r
:IiI~t
;T~k'~2th~ mlody I t ~ , yOU ! ~ L3 ~2:=2 [j L'~ O !
ran ~ l D ~ t t h ~ matt o n t i m
C\~ 0 0
r4~d'lr
~ ~ 1 . ~ , . _. l ~ User's Favorite I
[ ST~g~ 3: ~,~ i........ ~ m 16-bars musical work I

INlv/Mt~{~" bJtt~ ]hs

~O~r e~l~k]n ~f th~

I~q~q~tf~n o f four 4"~zir
mml , ~ c ~ h ~ l o n ~ i thn~e
~tlan ~', . . . . . . . . . . . . . . . . ~:~.::...~.~:.:
Fig. 3 User Interface

(1) Evaluation of Work: Users evaluate impressions of the works by choosing one
from "Good", "Very Good", "Neutral" or "Bad".
(2) Partial Evaluation of Work: If users find good parts in the melody line, users
point them out with "Good Part" button that is displayed above the melody
line.
(3) Copying and Composing a 16-Bar Work: Users compose a favorite 16-bar
musical work in the "Users Favorite" area, connecting four 4-bar works. If users
find a favorite work, they copy it into the area. If users newly find better works,
preserved works are replaced by them.
(4) Evaluation of 16-Bar Work: If four favorite 4-bar musical works are preserved,
the composed 16-bar work is evaluated every three generations. Users evaluate
the work whether the work is good or not as a whole with a 5-point scale, +2
(very good connection), ~ -2 (very bad connection).

4.4 GA Operators and Generation of New Musical Works

The system gives GA operators to chromosomes in the pools and generates
new 4-bar musical works through the following procedures.
(1) Selection of Parent Chromosomes: Fifty chromosomes are selected from each
pool as the next generation parent chromosomes. Selection criteria are as fol-
lows. a) A chromosome with a "Very Good" or "Good" evaluation is copied
fivefold. A chromosome with a "Neutral" evaluation is copied once. b) A chro-
mosome with a "Very Good" or "Good" evaluation in other pools is copied once,
respectively, c) The chromosomes in the users favorite area are copied ( 5 + a ) -
fold, where a is defined by the overall evaluation of the 16-bar work; "Very
Music Composition by Interaction between Human and Computer 187

Good", "Good", "Neutral", "Bad", or "Very Bad" corresponds to +4, +2, 0,

-2, or -4, respectively, d) Ten chromosomes are newly generated, e) Deficiencies
are chosen from the original pool at random, and 50 chromosomes are preserved
in all at the current generation. These selection procedures are repeated for the
1st through 4th pools. The parameter values, e.g., fivefold copying, (5+~)-fold
copying, are determined after trial and error.
(2) Crossover and Mutation: Examples of crossover and mutation are shown in
Fig. 4. Crossover is applied to 70 % of chromosomes. Crossover is defined as the
copy operation of parts of a melody line from one chromosome to another one,
which are chosen at random. Mutation is applied to 30 % of the chromosomes
after crossover. Mutation is defined as the re-construction of parts of the melody
line. The probabilities of crossover and mutation are defined by trial and error.
I
Copyingpartsof melody(1-4Mocks vath samesizeand chord) ' "Good part" evaluation toward parts in the rn~odyli~ byusers
I ..... ?. . . . . . . . . . . . .
ChromosomeA I I L+"= ' ' ~++ ,i I ] I '] I I I' I I I I I,I I I'. I +I I I I',

[ l I I I I t I i[ . . . . O...... ~ Newlv~G
....... /
I ~ses ~ -~fge m

NewChmmosome I~M I ~ I I ! 4~ I

~
),
R...... tructlonof. . . . partsofch. . . . . . . . <+0++)+I..,/l
, (. . . . ~
......... /I\
,,~ /I\ /][\
+IX+ /]~ //~
G ..... rio. and replacement of memdy 0"4 bloc~: i, l+
Copyir~ Virus, ,o Ch ..... +++ +I
i I I I I I [ I [ [ I I I I I ] I I: i .... I I I I

I I I I ~'~v!":~a I i I I I [ II I t I
=
Fig. 4 GA Operators: Crossover, Mutation, Virus Genera-
tion, and Virus Infection

(3) Virus Generation and Virus Infection The partial evaluation of the works
is reflected in the GA operators named virus generation and virus infection,
based on the virus evolution theory, as shown in Fig. 4. The virus generation is
defined as the preservation of parts of the melody line evaluated partially "good"
in the viruses database. The virus infection is defined as a copy operation of
viruses to the chromosomes in the pool. The usefulness of virus generation and
virus infection is confirmed by the previous paper. 1~> The questionnaire results
answered by the subjects who compose musical works show that the partial
evaluation of a melody line is reflected well in the melody line of the musical
work generated in the next generation.

w Evaluation Experiments
To confirm the validity of the present system, 10 subjects, who are from 20
years old to 30 and have no music composition experience, carry out evaluation
experiments.
Experiment A: Experiment A is carried out to confirm whether the system gener-
ates music reflecting users' evaluation, and whether the system generates music
with various kinds of melody lines reflecting users' subjectivity toward music.
Subjects try to compose their favorite "cheerful musical works" using the present
system. The first generation chromosomes have the same data among all sub-
188 M. Unehara and T. Onisawa

i:i :/ i 84 84184184184
~,
::. :: :::: : ::

I
Evaluation of the works
. . . .

.--,e--- by subjects (Average)

.... 4 .... Evaluation of the works
by others (Average)

3 6 9 12 15 Generation

Fig. 5 Evaluation of Musical Works

jects. The subjects compose music with the system until the 15th generation.
After the composition, the works composed at the 3rd, 6th, 9th, 12th, and 15th
generations by the subjects themselves are presented to the subjects in random
order. Each subject listens to and evaluates those works composed by the sub-
ject himself/herself with a 5-point scale, where 5 point is the best and 1 point
is the worst.
Experiment B: Experiment B is carried out in order to evaluate composed mu-
sical works from the objective point of view. Other twelve subjects, who don't
compose music in experiment A, listen to the works composed in the 3rd, 6th,
9th, 12th, and 15th generations by some subject in experiment A in random or-
der. The twelve subjects rate the works with a 5-point scale, where the 5 point
means that they feel the most cheerful and the 1 point means that they feel the
least cheerful. Then, the subjects listen to and evaluate other works composed
by another subject. The twelve subjects repeat these evaluations of the works
composed in the 3rd, 6th, 9th, 12th and 15th generations in experiment A.
Results and Remarks: Figure 5 shows the average of subjects' evaluations of
composed works at every three generations. The solid line expresses the results
of experiment A, and broken line expresses the results of experiment B. It is
found that both of the values increase with the progress of generations. In the
result of experiment B, a t test is conducted to confirm whether there was a
significant difference between the average value at the 3rd generation and that
at the 15th generation. The t value was 3.93, larger than t(218,0.05) = 1.97.
Therefore, it is confirmed that composed works also become more cheerful even
for subjects who don't compose them. However, the solid line becomes better
than the broken line. From this point of view, it is found that the composers are
more satisfied with their works than other subjects who don't compose them.
Figure 6 shows examples of the melody part scores composed by two subjects.
Although the subjects try to compose cheerful music and start with the same
Music Composition by Interaction between Human and Computer 189

'~t I ' Ii I I ~" r i, ~ I

. r, t. . j :# . . ~. P I
.~4.........................q ......:.................~.-.-4........f...........t..............] ............ ~'-'~-- ~ " ~ "- - u ~ ' ~

~,~ J i i.~5 ;'i" i( r f , : t

Example A Example B

Fig. 6 Example of Composed Melody Part Scores

chromosomes, it is found that various works with different melody lines are
generated, reflecting their subjective evaluations toward music. These results
show that the system can compose music reflecting each user's feeling of music.

w Summary
This paper describes a music composition system, by which 16-bar mu-
sical works reflecting users' feeling of music are composed. An Interactive GA
is introduced to the system so that users' subjectivity toward music is reflected
in composed music. One chromosome corresponds to a 4-bar musical work,
and the GA operators are applied to chromosomes according to users' subjec-
tive evaluations. These chromosomes are gradually evolved over the progress
of generations, aiming to generate users' favorite musical works. From the ex-
perimental results, it is found that various works are composed and that the
evaluation to the works increased over the progress of generations. T h a t is, the
system can compose music reflecting users' feeling of music.
Future research aims to construct a system that composes not only the
melody part but also the backing parts completely.

References
1) Biles, J. A., "GenJam: Interacting with a musical IGA," Proceedings o f the In-
ternational Conference on Systems, Man, and Cybernetics 1999, Tokyo, Japan, 3, pp.
652-656, 1999.
2) Igarashi, S., Scientific Analysis o f Music Performance, YAMAHA Music Media Co.,
Tokyo, Japan, 2000 (In Japanese).
3) Johanson, B. and Poll, R., "GP-Music: An Interactive Genetic Programming
System for Music Generation with Automated Fitness Raters," Proceedings o f
the Third Annual Conference on Genetic Programming, MIT Press, 1998.
4) Katayose, H. and Inokuchi, S., "Computer and Music," Journal o f the Institute
o f Electronics, Information and Communication Engineers, Tokyo, Japan, 77, pp.
767-770; 1994 (In Japanese).
5) Kitagawa, Y., Handbook o f Chord Progression, RittorMusic, Tokyo, Japan, 1999
(In Japanese).
6) Kitagawa, Y., Handbook of Music Theory, RittorMusic, Tokyo, Japan, 1999 (In
Japanese).
190 M. U n e h a r a a n d T . O n i s a w a

7) Nelson, G. L., "Sonomorphs: an Apprication of Genetic Algorithms to Growth

and Development of Musical Organisms," Proceedings of the Fourth Biennial Art &
Technology Symposium, Conneticut College, USA, pp. 155-169, 1999.
8) Onisawa, T., "Soft Computing Technique in Kansei (Emotional) Information
Processing," in Soft Computing and Human-Centered Machines, Liu, Z.Q. and
Miyamoto, S. (eds), Springer-Vertag, pp. 215-248, 2000.
9) Roads, C., Computer Music, Tokyo Denki University Press, 2001 (In Japanese).
10) Takagi, H., "Interactive Evolutionary Computation: Fusion of the Capabilities
of EC Optimization and Human Evaluation," Proceedings of the IEEE, 89, 9, pp.
1275-1296, 2001.
11) Taniguchi, T., Music and Emotion, Kitaohji-Shobo, Tokyo, Japan, 1998 (In
Japanese).
12) Tokui, N., "Music Composition with Interactive Evolutionary Computation," in
Proceedings of 3rd International Conference on Generative Art, Iba, H. (eds), Milan,
2000.
13) Umemoto, T., Study on Musical Psychology, Nakanisiya Shuppan Co., Tokyo,
Japan, 2000 (In Japanese).
14) Unehara, M. and Onisawa, T., "Composition of Music Using Human Evalu-
ation," Proceedings of 2001 IEEE International Conference on Fuzzy Systems, Mel-
bourne, Australia, 2001.
15) Unehara, M. and Onisawa, T., "Music Composition System with Human Eval-
uation as Human Centered System," International Journal of Soft Computing,
Springer-Verlag, 7, pp. 167-178, 2001.
16) Unemi, T., "A Tool for Composing Short Music Pieces by Means of Breeding,"
in Proceedings of the IEEE lnternational Conference on Systems, Man and Cybernetics,
Nakada, E. (eds), pp. 3458-3463, 2001.
17) Wiggins, G., et.al., "Evolutionary Methods for Musical Composition," Interna-
tional Journal of Computing Anticipatory Systems, 1998.
18) Yahagi, H., Electone Backing Pattern Book, YAMAHA Music Media Co., Tokyo,
Japan, 1999 (In Japanese).
19) YAMAHA Co., Easy Composition Master, YAMAHA Co., Tokyo, Japan, 1999.

Muneyuki Unehara: He received his M.S. in Engineering in 2002

from Institute of Science and Engineering, University of Tsukuba.
Currently, he is a Ph.D. candidate of Graduate School of Sys-
tems and Information Engineering, University of Tsukuba. His
research interests include the construction of intelligent systems
by considering soft computing techniques and human interface.
Music Composition by Interaction between Human and Computer 191

Takehisa Onisawa, Ph.D.: He received Dr.Eng. in Systems Sci-

ence in 1986 from Tokyo Institute of Technology. Currently, he
is a Professor in the Graduate School of Systems and Informa-
tion Engineering, University of Tsukuba. His research interests
include applications of soft computing techniques to human cen-
tered systems thinking. He is a member of IEEE and IFSA.