Evolving Kinship in Artificial Neural Networks

Angela Mastrianni
Adviser: Prof. Douglas Blank

April 25 2016

Abstract

The goal of this thesis is to explore how evolutionary constraints and kinship will affect
the performance of artificial neural networks. Our hypothesis is that a population of artificial
neural networks will be able to evolve kinship-like behaviors. This hypothesis is based off the
discussion in a paper entitled “Cooperation and the Evolution of Intelligence” by McNally, et
al. [9]. To test our hypothesis, we apply the neural networks to two game theory problems: the
Iterative Prisoners’ Dilemma and the Iterative Snowdrift Dilemma. The neural networks are
evolved with two evolutionary methods– one with fixed topologies and memory, called CONX
and one with evolving topologies and the potential for memory, called NEAT. We found that
the populations of artificial neural networks evolved to have kinship properties with both NEAT
and CONX, and that they could differentiate their strategies for playing against kin and non
kin.
Keywords: Artificial Neural Networks (ANNs), Evolutionary Algorithms, Kinship, NEAT,
CONX, Cooperation, Intelligence

1
1 Introduction

One goal of cognitive science is to attempt to model human behaviors with artificial neural networks,

which are computational models of the human brain. One area of expanding research is investigating

if artificial neural networks (ANNs) are capable of cooperating with other artificial neural networks.

For example, Yong and Miikkulainen discovered that ANNs could be evolved in separate sub-

populations and still cooperate to capture prey [16] . Potter, et al. found that ANNs would take

on different roles and cooperate in order to herd “sheep” and protect them from “foxes” [13] .

However, in these situations, all of the ANNs received direct benefits from cooperation and, thus,

had incentives to cooperate.

McNally, et al. investigated cooperation between individuals when they participate in tasks

where cooperation does not always give the maximum payoff for the individual [9]. To accomplish

this, they evolved ANNs with genetic algorithms as they played each other in two game theory

problems– the Iterative Prisoner’s Dilemma and the Iterative Snowdrift Dilemma, described in

section 1.4. In both problems, the best scenario for the individual occurs when they defect and

their opponent cooperates. The researchers looked at the frequency of cooperation and the differ-

ent behavioral strategies that emerged as the intelligence of the neural networks increased while

participating in the social dilemmas. The researchers defined intelligence as the number of neurons

in the neural network– a neural network with more neurons would be considered more intelligent.

In the Iterative Prisoners’ Dilemma, as intelligence increased, cooperation decreased as the neu-

ral networks entered in “arms races” with one another, trying to defeat the other individuals in

the population. In the social dilemma that encouraged cooperation more, the Iterative Snowdrift

Dilemma, cooperation slightly increased as intelligence increased.

For future research, McNally, et al. suggested looking at the role “kin” selection plays in the

evolution of intelligence and cooperation. According to Hamilton’s theory of kin selection [5], an

organism will cooperate with a related organism at a cost to themselves since helping that related

organism will increase that organism’s fitness and therefore, help the chances of their shared genes

being passed down to the next generation. A study done by Madsen et al. showed that humans

in two different cultures act more altruistically to related individuals even if it comes at a high

2
physical cost [7]. This study helped proved Hamilton’s theory of kin selection in humans. McNally,

et al. hypothesized that “...high relatedness may hinder the evolution of intelligence by driving

unconditional cooperation to fixation in the population” [10]. If kinship levels are too high in a

population, there may not be the same arm races in the population that lead to the development

of intelligence. They also state that “while artificial neural networks are not real brains, relying

on abstraction of the activity of millions of real neurons down to a manageable number of artificial

neurons, they can provide insight into the dynamics of cognitive evolution and allow for the flexible

evolution of behaviour” [11]. The goal of this thesis is to explore the effects of using different

methods to evolve neural networks in game theory problems and examine if kinship-like behaviors

can be generated.

1.1 Artificial Neural Networks

To understand artificial neural networks (ANNs), it is first important to understand how the human

brain works, as ANNs are modeled on the structure of the human brain. The brain consists

of neurons which send electrical impulses to nearby neurons. The activity level of a neuron is

measured by the average number of times that they fire an electrical impulse. The brain consists

of 1011 neurons which are arranged in a maximum of six layers of connectivity, with not every

connection going through all layers. Synaptic gaps separate the neurons. Signals going across the

gaps either excite or inhibit the electrical potential of the receiving neuron. When the electrical

potential of the receiving neuron reaches a certain threshold, it will fire [14].

1.1.1 Structural Explanation of ANNs

ANNs are composed of two elements: units (which correspond to neurons) and connection weights

(which correspond to synapses) between the units. The activation weight of a unit parallels the

firing rate of a neuron while the connection weights between neurons parallel neurons potential

to excite nearby neurons. Real numbers are used to represent the activation weights of units and

the connection weights between units. The set of units and connection weights are arranged in a

certain topology to create the ANN. Generally, the topologies will have a layer of input node(s),

3
which take in information from the environment. There is also a layer of one or more output nodes

which return information back to the environment. Finally, there is usually also one or more layers

of hidden nodes which only interact with other hidden nodes, input nodes or output nodes, not the

outside environment. Some neural network topologies remain static, while others have the ability

to evolve, gaining or losing units and connection weights over time [14]. A sample neural network

is shown in figure 1. There are two ways to determine the appropriate weights for neural networks,

through learning or through evolution.

1.1.2 Functional Explanation of ANNs and Learning

ANNs are used to model human brains. They take in information from the outside environment

as input and then use this information to make decisions, returning the decisions as output. For

example, consider the XOR problem. In this problem, the neural network would take two inputs-

each of which is a 0 or 1. A neural networks then uses these inputs to output a 1 or 0. There are

different types of learning for ANNs: supervised, unsupervised and reinforcement learning. The

XOR problem could use supervised learning. In supervised learning, the neural network learns if

their output was “right” or “wrong”, and the correct output if they made the wrong one. Assume

the neural network receives a 0 and 1 input and outputs a 0. It will then receive feedback that it

made the wrong output and should adjust weights. In unsupervised learning, the ANN receives no

feedback on their decisions but instead looks for recurring patterns from the input. For example,

if an ANN was travelling around a simulated maze, they may learn to correlate red walls with long

paths. In reinforcement learning, the ANN will receive the occasional reward or punishment for its

decisions. If the ANN was travelling around the maze, they would occasionally receive input as to

how close they were to the goal point of the maze. It is important to find the right rate of learning.

If the ANN learns too “quickly” it may jump between potential solutions missing the right one.

However, if an ANN learns slowly, it can take a long time to find the right solution [3].

4
Figure 1: A Sample Neural Network: Green nodes are input nodes, grey nodes are hidden nodes,
the black node is a bias node and the blue node is an output node.

1.2 Evolutionary Algorithms

It is important to see how evolution works in biology in order to understand how it can be applied

to computer problems. In evolution, the “fittest” species in a population are the ones most likely

to live long enough to reproduce and pass their genes down to their offspring, a process called

natural selection. According the the basic principles of biology, every organism has cells which

consist of chromosomes, or strands of DNA. Chromosomes are divided into genes which encode

particular proteins. Humans have 23 pairs of chromosomes in each cell. In sexual reproduction, each

parent’s pairs of chromosomes exchange genes to produce a single chromosome, a process known as

crossover or recombination. The offspring then receives 23 chromosomes from each parent to have

23 pairs. Other organisms that do not have pairs of chromosomes (known as haploids) reproduce by

exchanging genes between each parent’s chromosomes. Mutation can occur when a gene is altered

in the offspring during reproduction [12].

Simple genetic algorithms encode chromosomes as bit strings, with genes being a block of bits in

the string. To mimic selection, the genetic algorithm will ensure that “fitter genes” are more likely

to be selected for reproduction by using a fitness function to measure the fitness of a chromosome

in its population. The genetic algorithm replicates crossover by picking a set point on the bit

strings of two genes and taking one set of bits before that point from one parent and the other set

5
of bits after that point from the other point to form the bit string of the offspring. Finally, the

genetic algorithm mimics mutation by randomly selecting a bit and switching it to the opposite

value. In a genetic algorithm, a population will be evolved for a certain number of generations,

with reproduction occurring at the end of each generation [12].

1.3 Methods for Creating and Evolving ANNs

There are different methods for creating and evolving ANNs. This research explores two different

methods. One method, the NeuroEvolution of Augmenting Topologies (NEAT) evolves ANNs with

changing topologies that have the potential to evolve memory if it’s advantageous. The other

method, CONX, evolves ANNs with fixed topologies and a set amount of memory.

1.3.1 NeuroEvolution of Augmenting Topologies

NeuroEvolution of Augmenting Topologies (NEAT) is a genetic algorithm created by Stanley and

Miikkulainen that allows for network topologies to evolve over time [15]. The algorithm works by

starting with a simple genome and then allowing that genome to complexify if needed. NEAT begins

with ANNs that have no hidden nodes and then adds or removes nodes later. NEAT’s algorithm has

three main components. The first component keeps track of genes that exist in genomes of different

sizes. To accomplish this, it assigns a global innovation number to every gene which indicates the

chronological time when that gene first appeared in the population. The second component divides

the population into different species, allowing ANNs of different sizes to remain. NEAT achieves

this by grouping networks of similar topologies into species and having networks within a species

compete with each other instead of with the entire population. Speciation is important as it allows

for new topological innovations to have time to optimize and not be thrown out immediately. The

third component creates an initial population of ANNs with small, uniform topologies.

Stanely and Miikkulainen tested NEAT’s abilities in the domain of a robot duel. They choose

a domain with coevolution. Coevolution leads to a continued arms race, necessitating networks

with complex structures. In the duel tasks, two robots move around in an arena with the goal of

overpowering the other robot. Whenever the robots collide, the robot with the higher energy level

6
wins. In their experiment, they evolved networks three different ways. They evolved networks using

NEAT, using a genetic algorithm that produced fixed topologies and using simplification– starting

with complex topologies, then allowing nodes and connections to be removed. They found that

neural networks built with NEAT had the highest average performance in the robot duels compared

to those built with fixed topologies or simplification. Figure 2 shows the changes in topology as the

network undergoes evolution with NEAT. The left picture is the network’s structure at the start

of evolution while the the picture on the right is the network’s structure at the end of evolution.

Figure 2: Evolution with NEAT (Beginning of evolution on the right and End of evolution on left).
Green nodes are input nodes, grey nodes are hidden nodes, the black node is a bias node and the
blue node is an output node.

1.3.2 CONX

CONX is a type of neural network structure created by Blank, Konolige, Kumar, Meeden and

Yanco [1]. CONX consists of different layers made up of nodes and connections between layers.

There are three different types of layers: input, output and hidden. Input and output layers interact

with the outside environment while hidden layers only interact with other hidden layers or input

and output layers. CONX is capable of storing the memory of the recent past. To accomplish this,

it implements simple recurrent networks (SRNs), which were developed by Jeffery Elman [4]. In

SRNs, every hidden layer has an associated context layer. The context layer is the same size as its

7
associated hidden layer and has weighted connections back to the hidden layer. After a network

completes propagation, the activation values from a hidden layer will be copied into the context

layer. These values can then be passed back to the hidden layer at a later time.

CONX also comes with a method for evolving populations of neural networks made with CONX.

The evolution is done via a genetic algorithm, where different fitness functions can be implemented

depending on the problem. However, unlike the evolution done by NEAT, the hidden layers and

connections can not be added or removed from the CONX topology during evolution. Thus,

throughout evolution, only the activation weights and connection weights can change, not the

topology of the network.

1.4 Game Theory Problems

The Prisoner’s Dilemma is a well-known problem in Game Theory . In the dilemma, two prisoners

are asked to indict the other in the crime. If they both indict each other, they will serve more time

than if neither indicts the other. If only one indicts the other, they will be set free while the other

will have to serve the maximum amount of time. In the dilemma, a player can choose between two

moves- cooperation or defection. The Iterative Prisoner’s Dilemma (IPD) consists of more than one

round of the Prisoner’s Dilemma between two players. The best scenario for the participant is that

they defect while their opponent cooperates. This results in costs to the opponent while benefiting

this participant. IPD represents a social dilemma as the mean payoff for total cooperation is higher

than the mean payoff for complete defection. The Iterative Snowdrift Dilemma (ISD) is a lesser

known problem in Game Theory. It represents situations where the costs of cooperation are shared

among cooperators. In the dilemma, two cars could be stuck behind on a snowbank. If both drivers

stay in the car, they are both stuck behind the bank. If one driver shovels out the car, they do all

the work while the other driver reaps the reward. If both drivers shovel, the car is dug out faster

than if only one person is shovelling and the effort is split among the two people. Like IPD, ISD is

also a social dilemma as a player will favor defecting while the other player cooperates, as seen by

the payoff matrix in 2.1 [9].

In both game theory problems, the best situation for a player is the same: they defect and their

8
opponent cooperates. However, the worst situation for a player differs between the games. In IPD,

the worst situation for a player is that they cooperate and their opponent defects. However, in ISD,

the worst situation for a player is that both they and their partner defect. Due to this difference,

ISD encourages cooperation more than IPD. Kummerli et al. compared human cooperation in IPD

to human cooperation in ISD [6]. They had 96 individuals play both IPD and ISD against other

human participants. They found that the frequency cooperation was higher in iterations of ISD

than in iterations of IPD. They also found that women were more likely to cooperate than men in

IPD. In their study of 96 players, 19 used the Tit-for-Tat behavioral strategy (defined in section

2.4), 32 used a Pavlovian strategy (defined in 2.4), and the rest of the 45 players had a undefined

strategy. Players with prior knowledge of the game were not more likely to use a particular strategy

than players with no prior knowledge of the game.

2 Methods for Evolution

2.1 Game Setup

We ran 10 experiments of the Iterative Prisoner’s Dilemma and 10 experiments of Iterative Snow-

drift Dilemma, evolving 10,000 generations of neural networks with NEAT and with CONX. Each

game consists of a population of fifty neural networks that play every other individual for fifty

rounds of the dilemma. In each round, each individual gets a payoff score based on their action

and their opponent’s action. We used the same payoff scores as the original study by McNally, et

al. The payoff scores for IPD are located in table 1 and the scores for ISD are located in table 2.

Cooperation Defection
Cooperation 6 7
Defection 1 2

Table 1: Payoff Scores for IPD

9
 Cooperation Defection
Cooperation 5 8
Defection 2 1

Table 2: Payoff Scores for ISD

2.2 Neural Network Setup for NEAT

The neural networks begin with three input nodes: the individual’s payoff score from the previous

round, their opponent’s score from the previous round and a bias. They have one output node,

which is the probability that they would cooperate in that round. Each network begins with one

hidden node and the networks could be recurrent. The networks evolved each generation with

NEAT.

Parameter Setting
Generations 10,000
Rounds 50
Population size 50
Input nodes 3
Output nodes 1
Prob. for recurrent link 0.25
Prob. to add link 0.03
Prob. to add node 0.01
Survival Rate 0.25
Crossover Rate 0.7
Overall Mutation Rate 0.25

Table 3: MultiNEAT parameter settings used in the NEAT experiments.

2.3 Neural Network Setup for CONX

The neural networks had two input nodes: the individual’s payoff score from the previous round

and their opponent’s payoff score from the previous round. They had three hidden nodes and one

context node associated with each hidden node. Finally, similarly to NEAT, they also had one

output node which was the probability that they would cooperate in that round.

10
 Parameter Setting
Generations 10,000
Rounds 50
Population size 50
Input nodes 2
Output nodes 1
Hidden Layer Nodes 3

Table 4: Parameter settings used in the CONX experiments.

2.4 Closest Pure Behavioral Strategy

McNally, et al used four pure strategies in their study to compare with their neural network

strategies. They were Always Cooperate, Always Defect, Tit-For-Tat and Pavlovian. In the always

cooperate and always defect, the pure strategy will either always cooperate or always defect. In

the Tit-For-Tat strategy, the player repeats whatever action their opponent choose in the previous

round. In the Pavlovian strategy, the player repeats a move if that move earned a payoff score

above a certain threshold in the previous round. We followed the same strategy for classifying

which pure strategy our networks were closest to as the study by McNally et al. We had each pure

strategy play itself and all other pure strategies, coding each move with a 1 for cooperation and a

0 for defection. We also had each network play against all 4 strategies, coding the moves in the

same manner. We then compared that sequence of moves to the sequences of moves between each

pure strategy with itself and the other pure strategies. The network would then be assigned to the

behavioral strategy that minimized the sum of squares between the sequences.

11
 Average Freq. of Cooperation
Lowest 1.80%
Highest 3.26%
Average 2.58%

Table 5: NEAT: The Lowest, Highest and Average Frequencies of Cooperation out of 10 Runs of
IPD

Freq. of Cooperation
Lowest 0.02%
Highest 49.74%
Average 3.33%

Table 6: NEAT: Frequency Distributions in a Run of IPD

3 Results

3.1 NEAT

Figure 3: NEAT: Cooperation Frequencies in IPD (a) and ISD (b)

12
 Average Freq. of Cooperation
Lowest 22.99%
Highest 24.83%
Average 23.89%

Table 7: NEAT: The Lowest, Highest and Average Frequencies of Cooperation out of 10 Runs of
ISD

Freq. of Cooperation
Lowest TBD%
Highest TBD%
Average TBD%

Table 8: NEAT: Frequency Distributions in a Run of ISD

Tables 5 and 7 show the runs with the highest and lowest average frequencies of cooperation of

IPD and ISD respectively. Detailed results from all ten runs can be found in the appendix. The

average frequency of cooperation over the ten runs was 2.58% in IPD while the average frequency

of cooperation over the ten runs in ISD was 23.89%, as highlighted by the tables. Figure 3 shows

graphs of the frequency of cooperation over the 10,000 generations for a run of IPD and a run of

ISD. As shown by the tables and graphs, the frequency of cooperation among the neural networks

was much higher in ISD than in IPD. The graph in Figure 3 also shows that cooperation levels

remain constant in ISD, as they did in the original study, by McNally et al [9]. Table 9 shows

the average amount of each behavioral strategies that the members of the population displayed

over the ten runs. In IPD, approximately 565 members of the 50,000 population had a strategy

closest to Always Cooperate, 49,220 had a strategy closest to Always Defect, 115 had a strategy

closest to Tit-for-Tat and 95 had a strategy closest to Pavlovian. In ISD, approximately 3,445 of

the population had a strategy closest to Always Cooperate, 44,710 had a strategy closest to Always

Defect, 1,020 had a strategy closest to Tit-for-Tat and 810 had a strategy closest to Pavlovian. This

table highlights how neural networks in ISD displayed a greater diversity of behavioral strategies

than neural networks in IPD. Full tables of the behavioral strategies in all ten runs for IPD and

ISD can be found in the appendix.

13
 Experiment Always Cooperate Always Defect Tit for Tat Pavlovian
IPD 1.13% 98.45% 0.23% 0.19%
ISD 6.91% 89.42% 2.04% 1.62%

Table 9: NEAT: Average Frequencies of Each Behavioral Strategy

3.2 CONX

Tables 10 and 12 show the runs with the highest and lowest frequencies of cooperation of IPD

and ISD respectively. Detailed results from all ten runs can be found in the appendix. The

average frequency of cooperation over the ten runs was 5.07% in IPD while the average frequency

of cooperation over the ten runs of ISD was 25.06%, as highlighted by the tables. Figure 4 shows

graphs of the frequency of cooperation over the 10,000 generations for a run of IPD and a run

of ISD. The average frequencies of each behavioral strategy over the generations for the ten runs

can be found in table 14. In IPD, approximately 1,765 members of the 50,000 population had a

strategy closest to Always Cooperate, 47,735 had a strategy closest to Always Defect, 335 had a

strategy closest to Tit-for-Tat and 185 had a strategy closest to Pavlovian. In ISD, approximately

7,515 of the population had a strategy closest to Always Cooperate, 40,780 had a strategy closest to

Always Defect, 860 had a strategy closest to Tit-for-Tat and 845 had a strategy closest to Pavlovian.

Like with networks in NEAT, neural networks with CONX in ISD displayed a greater diversity of

behavioral strategies than neural networks in IPD. Full tables of the behavioral strategies in all ten

runs for IPD and ISD can be found in the appendix.

Average Freq. of Cooperation

Lowest 3.92%
Highest 6.44%
Average 5.07%

Table 10: CONX: The Lowest, Highest and Average Frequencies of Cooperation out of 10 Runs of
IPD

14
 Freq. of Cooperation
Lowest 0.0%
Highest 64.96%
Average 5.38%

Table 11: CONX: Frequency Distributions in a Run of IPD

Average Freq. of Cooperation

Lowest 22.78%
Highest 31.57%
Average 25.06%

Table 12: CONX: The Lowest, Highest and Average Frequencies of Cooperation out of 10 Runs of
ISD

Figure 4: CONX: Cooperation Frequencies

15 in IPD (a) and ISD (b)
 Freq. of Cooperation
Lowest 2.89%
Highest 61.71%
Average 24.74%

Table 13: CONX: Frequency Distributions in a Run of IPD

Experiment Always Cooperate Always Defect Tit for Tat Pavlovian

IPD 3.53% 95.47% 0.67% 0.37%
ISD 15.03% 81.56% 1.72% 1.69%

Table 14: CONX: Average Frequencies of Each Behavioral Strategy

4 Methods for Kinship Extension

We extended the experiments with CONX and NEAT to see if neural networks could learn to

cooperate with members of the population that were their kin. To accomplish this, we divided the

population in two groups. The first twenty-five members of the population were related to each

other and the second twenty-five members of the population were completely independent (related

to no one else in the population). When a member of the kin group played another member of

the kin, they split the sum of their fitness scores, in order to simulate Hamilton’s theory of kin

selection. When two unrelated networks played each other, they each kept their own fitness score.

We ran two different kinship extensions– one with explicit kin inputs and one without explicit

kin inputs. When there was an explicit kin input, the structures of the neural networks in both

CONX and NEAT slightly changed. Each neural network had an additional input which would

be a 0 if they were unrelated to their opponent or a 1 if they were related. In the experiments

without the explicit kin input, there was no input indicating if an opponent was kin or not. We

ran 10 runs of up to 1,000 generations of IPD and ISD with this extension. We also looked at

the intelligence of the members of the kin group versus the members of the non-kin group. In the

original study done by McNally, et al. they measure intelligence as the number of neurons in the

ANN [9]. However, we thought this was a flawed measure since bigger brains do not always equate

to higher intelligence. It does not just matter how many resources one has, but also if they use

them effectively. We decided that it would be better to determine an individual’s intelligence based

16
on its performance against different types of opponents in the games. To measure intelligence, we

played each individual against players following the four pure strategies (Always Cooperate, Always

Defect, Tit-for-Tat and Pavlovian) and a player that just played random moves. An individual’s

intelligence was then their total payoff score from these five games.

Cooperation Defection
Cooperation 6 7
Defection 1 2

Table 15: Payoff Scores for Non-Kin in IPD

Cooperation Defection
Cooperation 6 4
Defection 4 2

Table 16: Kinship Payoff Scores for Kin in IPD

Cooperation Defection
Cooperation 5 8
Defection 2 1

Table 17: Payoff Scores for Non-Kin in ISD

Cooperation Defection
Cooperation 5 5
Defection 5 1

Table 18: Payoff Scores for Kin in ISD

5 Results for Kinship Extension

5.1 Explicit Kin Inputs

Tables 19-26 highlight the results for the kinship extension with an explicit kin input. We looked at

kin individuals cooperation with other kin members versus their cooperation with non kin members.

As shown by table 19, individuals in the kin who were evolved with NEAT cooperated with other

17
kin members at an average rate of 94.07% and cooperated with non kin members at an average rate

of 6.27% in IPD over the ten runs. In ISD, kin individuals evolved with NEAT cooperated with

other kin members at an average rate of 83.69% and non kin members at a rate of 31.10%, over the

ten runs, as shown by table 21. On the other hand, individuals in the kin who were evolved with

CONX cooperated with other kin members at an average rate of 72.59% and cooperated with non

kin members at an average rate of 29.98% in IPD, as shown by table 23. In ISD, kin individuals

evolved with CONX cooperated with other kin members at an average rate of 75.03% and non

kin members at a rate of 33.62%, as shown by table 25. Figures 4 and 5 show the changes in

cooperation by kin individuals evolved with NEAT over the first 100 generations of a run of IPD

and a run of ISD respectively. Figures 6 and 7 show the changes in cooperation by kin individuals

evolved with CONX over the 1,000 generations of a run of IPD and a run of ISD respectively. In all

figures, members of the kin start by displaying similar cooperation frequencies with kin and non-

kin. However, as the generations continue, kin begin to cooperate more with other kin members

and less with members outside their kin. Tables 27 and 28 show the payoff scores, which represent

intelligence, for the kin and non kin groups evolved by NEAT in ISD and IPD. There is no definitive

difference in the payoff scores between the kin and non-kin groups.

Run Kin Non Kin

1 94.16% 8.91%
2 95.78% 3.89%
3 94.35% 4.05%
4 93.90% 5.47%
5 91.40% 4.78%
6 93.72% 9.32%
7 94.72% 13.19%
8 92.74% 4.94%
9 95.03% 3.88%
10 94.94% 4.24%
Average 94.07% 6.27%

Table 19: NEAT with Explicit Input: Average Cooperation Frequencies Between Kin and Non Kin
in Runs of IPD

18
 Type Highest Lowest Average
Kin 99.98% 42.09% 94.85%
Non Kin 50.34% 0.068% 4.37%

Table 20: NEAT with Explicit Input: Frequency Distributions for Kin and Non Kin in a run of
ISD

Run Kin Non Kin

1 80.80% 36.42%
2 84.02% 33.21%
3 85.54% 30.99%
4 81.81% 35.34%
5 85.20% 31.90%
6 83.96% 31.60%
7 80.79% 32.00%
8 88.11% 25.28%
9 81.92% 21.90%
10 84.76% 32.40%
Average 83.69% 31.10%

Table 21: NEAT with Explicit Input: Average Cooperation Frequencies Between Kin and Non Kin
in Runs of ISD

5.2 No Explicit Kin Inputs

Tables 29-32 highlight the results for the kinship extension without an explicit kin input. We

looked at kin individuals’ cooperation with other kin members versus their cooperation with non

kin members when they had no explicit input indicating if an opponent was kin. As shown by

table 29, individuals in the kin who were evolved with NEAT cooperated with other kin members

at an average rate of 76.05% and cooperated with non kin members at an average rate of 51.21%

in IPD over the ten runs. However, some runs were more successful than others. In one run, they

cooperated with kin at rate of 68.14% and non kin at a rate of 38.96%. In ISD, kin individuals

evolved with NEAT cooperated with other kin members at an average rate of 64.76% and non

kin members at a rate of 42.65%, over the ten runs, as shown by table 30. On the other hand,

individuals in the kin who were evolved with CONX cooperated with other kin members at an

average rate of 74.38% and cooperated with non kin members at an average rate of 35.01% in

IPD, as shown by table 31. In ISD, kin individuals evolved with CONX cooperated with other kin

19
 Type Highest Lowest Average
Kin 99.96% 36.22% 83.82%
Non Kin 50.82% 11.11% 33.27%

Table 22: NEAT with Explicit Input: Frequency Distributions for Kin and Non Kin in a run of
ISD

Run Kin Non Kin

1 83.64% 25.83%
2 79.69% 60.90%
3 70.80% 17.71%
4 64.16% 22.00%
5 70.23% 26.03%
6 73.51% 21.90%
7 70.35% 15.00%
8 57.15% 60.08%
9 79.03% 22.64%
10 77.38% 27.03%
Average 72.59% 29.98%

Table 23: CONX with Explicit Input: Average Cooperation Frequencies Between Kin and Non Kin
in Runs of IPD

members at an average rate of 70.34% and non kin members at a rate of 34.96%, as shown by table

32. Figures 13 and 14 show the changes in cooperation by kin individuals evolved with NEAT over

the first 1,000 generations of a run of IPD and a run of ISD respectively. Figures 11 and 12 show the

changes in cooperation by kin individuals evolved with CONX over the 1,000 generations of a run

of IPD and a run of ISD respectively. In all figures, members of the kin start by displaying similar

cooperation frequencies with kin and non-kin. The figures show that the networks evolved with

NEAT have a harder time differentiating betwee kin and non kin when they are evolved without

the explicit kin input. The differentiation between kin and non kin is more present in the networks

evolved with CONX as compared to the networks evolved with NEAT, as also highlighted by the

figures.

20
 Type Highest Lowest Average
Kin 93.28% 26.18% 73.68%
Non Kin 71.22% 5.92% 30.09%

Table 24: CONX with Explicit Input: Frequency Distributions for Kin and Non Kin in a run of
IPD

Run Kin Non Kin

1 71.96% 37.58%
2 82.61% 34.80%
3 72.35% 31.04%
4 72.09% 29.74%
5 78.95% 43.36%
6 75.00% 27.25%
7 70.05% 37.41%
8 70.30% 28.27%
9 78.20% 34.64%
10 78.75% 32.10%
Average 75.03 33.62

Table 25: CONX with Explicit Input: Average Cooperation Frequencies Between Kin and Non Kin
in Runs of ISD

6 Discussion

6.1 Cooperation in Games without Kinship Components

The first set of findings comes from the experiments done with NEAT and CONX without any

kinship components. In these experiments, we co-evolved ANNs to 10,000 generations in IPD and

ISD. One finding from these results is that the ANNs display similar behaviors in IPD and ISD

to humans. As discussed earlier, Kummerli et al. found that cooperation between humans was

higher in ISD than IPD [6]. Tables 5, 7, 10 and 12 show that cooperation between neural networks

evolved with NEAT and CONX is higher in ISD than IPD. This is likely to due to the differences

in payoff structures between IPD and ISD. The worst case scenario for a player in IPD is that they

cooperate while their opponent defects. On the other hand, the worst case scenario for player in

ISD is that both they and their opponent defect, suggesting that ISD encourages cooperation more

than IPD. We also found that the ANNs evolved very basic behavioral strategies in these games.

21
 Type Highest Lowest Average
Kin 96.5% 43.21% 74.55%
Non Kin 87.31% 12.97% 36.98%

Table 26: CONX with Explicit Input: Frequency Distributions for Kin and Non Kin in a run of
ISD

Run Kin Non Kin

Lowest 149.99 178.142
Highest 179.29 179.142
Average 177.343 178.560

Table 27: NEAT with Explicit Input: Total Pay of Scores for the Kin Group and the Non-Kin
Group in Runs of IPD

The most popular strategy was Always Defect, and the second most common strategy was Always

Cooperate.

6.2 Cooperation between Kin and Non-Kin With Explicit Kin Inputs

The next set of results comes from the set of experiments where we divided the population into kin

and non-kin, giving an explicit kin input at every round of the game. We found that the ANNs

evolved to have high levels of cooperation with their kin, as shown by tables 19-26. Networks

evolved with both NEAT and CONX were able to differentiate between kin and non-kin games and

adjust their strategies accordingly. The networks with the potential for kin would cooperate heavily

with their kin but not with their non-kin. We found that there was no difference in intelligence

between networks that evolved playing against kin and non-kin and networks that evolved playing

only against non-kin.

We wanted to see if memory was necessary for this task. As shown by figures 5 and 6, NEAT

learns to differentiate between kin and non-kin in the first 10 generations of evolution. Figures 15

and 16 show the total number of recurrent nodes across ANNs evolved with NEAT in IPD and ISD

throughout 1,000 generations. As the networks do not many, if any, recurrent nodes when they

learn to differentiate between kin and non-kin, recurrent nodes are not needed for this task. To see

if CONX required memory, we evolved networks with CONX that did not have a memory bank for

22
 Run Kin Non Kin
Lowest 132.5 135.875
Highest 148.875 150.125
Average 142.450 142.961

Table 28: NEAT with Explicit Input: Total Pay of Scores for the Kin Group and the Non-Kin
Group in Runs of ISD

Figure 5: NEAT with Explicit Input: Kin vs. Non Kin Cooperation Frequencies in IPD

each hidden node. The topology remained the same as before, but there was no memory involved.

Networks evolved under these conditions also learned to differentiate between kin and non-kin, as

shown by figures 13 and 14.

We also believe that CONX’s banks of memory for each node can actually hinder its ability to

differentiate between kin and non-kin with the explicit inputs. Figures 7 and 8 highlight the fact

that, in some runs, networks evolved in CONX can take longer to learn to differentiate between kin

and non-kin than networks evolved with NEAT or networks evolved with CONX without memory.

The CONX method uses a set topology and searches through the state space of weights to find

the best weight configuration for the network in the particular problem. On the other hand, the

NEAT method uses an evolving topology, so it searches through the state space for both weights

and topology configurations for the problem. NEAT starts with no memory capabilities, but has

the potential to evolve memory over time, if memory is beneficial. At the start of evolution, NEAT

23
 Figure 6: NEAT with Explicit Input: Kin vs Non Kin Cooperation Frequencies in ISD

networks are smaller that CONX networks. The CONX networks also have a bank of memory for

each hidden node. We believe that this huge amount of memory could actually hinder CONX’s

ability to evolve networks that can differentiate between kin and non-kin games. This amount of

memory could make it harder for CONX to focus on the kin/non-kin input and learn to differentiate

between kin and non-kin games, at least in the beginning of evolution.

6.3 Cooperation between Kin and Non-Kin without a Explicit Kin Input

In the next of set of experiments, we ran the same kin experiments but removed the explicit kin

input to see if ANNs could still learn to differentiate between kin and non-kin without the explicit

input. The results from these experiments show that it is a much harder task for networks evolved

with NEAT to learn to differentiate between kin and non-kin when they do not have the explicit

kin input. This is especially true for ANNs evolved in the IPD games– in some runs, the networks

were not able to learn this differentiation in the 1000 generations. Figures 9 and 10 show how

the differentiation between kin and non-kin in networks evolved with NEAT is not as clear as the

differentiation when the networks were evolved with the explicit input. It makes sense that it would

take longer for ANNs evolved with NEAT to learn kin from non-kin without the explicit input since

it takes several generations to even start evolving recurrent nodes. However, CONX continues to

24
 Figure 7: CONX with Explicit Input: Kin vs. Non Kin Cooperation Frequencies in IPD

perform fairly well, even without the explicit kin input. This could be due to the fact that CONX

has memory banks from the start of evolution and does not have to evolve memory like NEAT.

7 Conclusions and Future Work

As discussed at the beginning of this paper, one expanding area of research in cognitive science

is investigating if ANNs will cooperate with one another. Our work led to three main findings–

all of which were related to this idea of cooperation between ANNs. Our first finding was that

ANNs cooperated more in the Iterative Snowdrift than the Iterative Prionser’s Dilemma. Similarly,

Kummerli et al. found that humans also cooperated more in the Snowdrift Dilemma than the

Prisoner’s Dilemma [6], as discussed earlier.

Our next two findings were related to the hypothesis by McNally et al. that kinship could be

involved in populations participating in the IPD and ISD. We found that given an explicit input

indicating if an opponent was kin, networks evolved with both NEAT and CONX could learn to

differentiate between kin and non kin, cooperating more with kin than non kin. In some instances,

it could take networks evolved with CONX a longer time to learn this differentiation. We speculated

that the large memory banks in CONX could be a hindrance– making it harder for the networks

25
 Figure 8: CONX with Explicit Input: Kin vs Non Kin Cooperation Frequencies in ISD

to focus on the kin/non-kin input. We also discovered that memory was not needed for this task

as networks evolved with CONX without memory also learned to differentiate between kin and

non-kin games. Lastly, we also found that networks evolved with NEAT and CONX could learn

to differentiate between kin and non kin when they were not given an explicit input indicating if

their opponent was kin. However, this was a harder task and not all runs of the experiment with

NEAT in IPD resulted in networks that learned this differentiation.

It’s important to look at the limitations with this research to see the possibilities for future

work. In this work, we engineer the definition of kinship. We specifically choose members of the

population to be related to each other and adjust their payoff scores to reflect Hamilton’s rule of

kin selection. Future work could look at creating a bigger artificial world with ANNs and seeing if

kinship would “fall out” of evolution, instead of being engineered into it. It would also be interesting

to see if other kinship principles could be created with ANNs. For example, the Cuckoo bird will

lay its eggs in another bird’s nest. Once the egg hatches, the other bird does not realize that it

is not its child, so it raises the baby bird as its own [2]. It would be interesting to see if ANN

could “fake” kinship in order to reap the rewards of the kinship without any costs. It would also

be interesting to see if evolution would then lead to the ANNs developing strategies to detect this

fake “kinship”.

26
 Run Kin Non Kin
1 62.12% 40.45%
2 74.70% 49.40%
3 82.75% 59.53%
4 78.30% 54.32%
5 82.24% 58.29%
6 77.11% 48.97%
7 76.68% 53.95%
8 68.14% 38.96%
9 82.39% 58.76%
10 76.06% 49.28%
Average 76.05% 51.21%

Table 29: NEAT without Explicit Input: Average Cooperation Frequencies Between Kin and Non
Kin in Runs of IPD

Overall, the goal of using artificial neural networks is model human behaviors. We showed that,

like humans, our artificial neural networks cooperated more in the Iterative Snowdrift Dilemma

than the Iterative Prisoners’ Dilemma. We also showed that the artificial neural networks, like

humans, were able to learn to differentiate between kin and non kin, adjusting their behavioral

strategies accordingly.

27
 Run Kin Non Kin
1 64.58% 42.89%
2 63.82% 43.11%
3 65.16% 43.15%
4 63.92% 41.45%
5 67.30% 43.55%
6 69.48% 43.71%
7 62.15% 42.20%
8 63.76% 41.67%
9 64.79% 42.22%
10 62.22% 43.55%
Average 64.76% 42.65%

Table 30: NEAT without Explicit Input: Average Cooperation Frequencies Between Kin and Non
Kin in Runs of ISD

Run Kin Non Kin

1 68.75% 19.11%
2 69.32% 20.76%
3 77.50% 52.70%
4 78.09% 53.51%
5 73.34% 37.20%
6 77.17% 26.76%
7 70.95% 16.43%
8 80.83% 69.34%
9 77.44% 28.27%
10 71.14% 26.00%
Average 74.38% 35.01%

Table 31: CONX without Explicit Input: Average Cooperation Frequencies Between Kin and Non
Kin in Runs of IPD

28
 Run Kin Non Kin
1 70.72% 30.96%
2 65.00% 33.43%
3 65.51% 37.72%
4 74.58% 44.00%
5 72.53% 34.32%
6 72.71% 32.11%
7 72.61% 36.51%
8 70.69% 36.62%
9 66.86% 30.71%
10 74.21% 33.53%
Average 70.34% 34.96%

Table 32: CONX without Explicit Input: Average Cooperation Frequencies Between Kin and Non
Kin in Runs of ISD

Figure 9: NEAT without Explicit Input: Average Cooperation Frequencies between Kin and Non-
Kin in IPD

29
Figure 10: NEAT without Explicit Input: Average Cooperation Frequencies between Kin and
Non-Kin in ISD

Figure 11: CONX without Explicit Input: Average Cooperation Frequencies between Kin and
Non-Kin in IPD

30
Figure 12: CONX without Explicit Input: Average Cooperation Frequencies between Kin and
Non-Kin in ISD

Figure 13: CONX without Memory: Average Cooperation Frequencies between Kin and Non-Kin
in IPD

31
Figure 14: CONX without Memory: Average Cooperation Frequencies between Kin and Non-Kin
in ISD

Figure 15: Total Number of Recurrent Nodes across all ANNs evolved with NEAT in IPD

32
Figure 16: Total Number of Recurrent Nodes across all ANNs evolved with NEAT in ISD

33
8 Appendix

Run Average Freq. of Cooperation

1 2.06%
2 2.23%
3 2.36%
4 2.31%
5 3.26%
6 3.21%
7 3.60%
8 1.80%
9 2.36%
10 2.58%
Average 2.58%

Table 33: Average Frequencies of Cooperation in Runs of IPD for NEAT (Data collected in Swarth-
more CS81 [8])

Run Average Freq. of Cooperation

1 23.67%
2 23.64%
3 24.76%
4 23.83%
5 24.46%
6 23.45%
7 23.89%
8 22.99%
9 24.83%
10 23.41%
Average 23.89%

Table 34: Average Frequencies of Cooperation in Runs of ISD for NEAT (Data collected in Swarth-
more CS81 [8])

34
 Run Always Cooperate Always Defect Tit for Tat Pavlovian
1 0.69% 98.99% 0.17% 0.14%
2 0.91% 98.68% 0.23% 0.17%
3 0.86% 98.68% 0.31% 0.14%
4 0.84% 98.75% 0.23% 0.18%
5 1.64% 97.91% 0.24% 0.21%
6 1.62% 97.92% 0.24% 0.22%
7 2.10% 97.40% 0.27% 0.25%
8 0.54% 99.21% 0.14% 0.11%
9 1.03% 98.56% 0.22% 0.19%
10 1.06% 98.41% 0.28% 0.26%
Average 1.13% 98.45% 0.23% 0.19%

Table 35: Average Frequencies of Each Behavioral Strategy in Runs of IPD for NEAT (Data
collected in Swarthmore CS81 [8])

Run Always Cooperate Always Defect Tit for Tat Pavlovian

1 7.01% 89.04% 2.21% 1.75%
2 8.19% 87.68% 2.25% 1.86%
3 6.13% 90.22% 2.07% 1.57%
4 6.97% 89.64% 1.93% 1.56%
5 6.19% 89.83% 2.26% 1.71%
6 7.01% 89.31% 2.03% 1.58%
7 6.84% 90.27% 1.58% 1.31%
8 6.78% 89.78% 1.94% 1.52%
9 6.49% 89.73% 2.09% 1.68%
10 7.44% 88.74% 2.08% 1.69%
Average 6.91% 89.42% 2.04% 1.62%

Table 36: Average Frequencies of Each Behavioral Strategy in Runs of ISD for NEAT (Data
collected in Swarthmore CS81 [8])

35
 Run Average Freq. of Cooperation
1 6.44%
2 3.92%
3 4.91%
4 4.20%
5 5.79%
6 6.26%
7 4.83%
8 4.53%
9 4.42%
10 5.39%
Average 5.07%

Table 37: Average Frequencies of Cooperation in Runs of IPD for CONX

Run Average Freq. of Cooperation

1 23.85%
2 31.57%
3 24.11%
4 24.80%
5 27.18%
6 25.20%
7 22.87%
8 22.78%
9 23.51%
10 24.74%
Average 25.06%

Table 38: Average Frequencies of Cooperation in Runs of ISD for CONX

Run Always Cooperate Always Defect Tit for Tat Pavlovian

1 4.78% 93.78% 1.00% 0.46%
2 2.78% 96.20% 0.82% 0.19%
3 3.63% 95.58% 0.61% 0.18%
4 3.32% 96.10% 0.43% 0.15%
5 3.80% 95.10% 0.73& 0.78%
6 4.22% 94.64% 0.69% 0.43%
7 3.03% 95.76% 0.74% 0.47%
8 3.20% 95.92% 0.52% 0.37%
9 2.83% 96.20% 0.54% 0.43%
10 3.69% 95.41% 0.61% 0.28%
Average 3.53% 95.47% 0.67% 0.37%

Table 39: Average Frequencies of Each Behavioral Strategy in Runs of IPD for CONX

36
 Run Always Cooperate Always Defect Tit for Tat Pavlovian
1 15.53% 81.16% 1.55% 1.75%
2 11.75% 84.72% 1.40% 2.14%
3 15.31% 81.21% 1.90% 1.52%
4 15.59% 80.32% 1.60% 2.49%
5 15.97% 79.51% 2.42% 2.10%
6 16.10% 79.34% 2.50% 2.02%
7 14.85% 82.71% 1.44% 0.98%
8 14.77% 82.73% 1.41% 1.08%
9 15.48% 81.92% 1.48% 1.11%
10 14.94% 81.97% 1.50% 1.59%
Average 15.03% 81.56% 1.72% 1.69%

Table 40: Average Frequencies of Each Behavioral Strategy in Runs of ISD for CONX

Figure 17: Frequencies of Behavioral Strategies in IPD (a) and ISD (b) for NEAT

37
9 Acknowledgements

A thank you to my peer, Lucas Franca do Nascimento Farias, for his collaboration on developing

code for the initial NEAT experiments in our CS81 Adaptive Robotics at Swarthmore, as well as

Professor Meeden for teaching the class. I would also like to thank Professor Cooper for reading

my drafts in senior seminar as well as my 2016 classmates for their contributions. Lastly, I would

like to thank Professor Xu for encouraging me to take Intoduction to Computing, thus starting me

on this path, and Professor Blank for serving as my adviser throughout this process.

38
References

[1] Blank, D., Konolige, K., Kumar, D., Meeden, L., and Yanco, H. Pyrobotics: Conx.

http://pyrorobotics.com/?page=Pyro. Accessed: 2016-04-24.

[2] Briggs, M. B. Cuckoos don’t sneak into other birds’ neststhey barge right in. The Smithso-

nian (2014).

[3] Downing, K. L. Intelligence Emerging: Adaptivity and Search in Evolving Neural Systems.

MIT Press, Cambridge, Massachusetts, 2015.

[4] Elman, J. L. Finding structure in time. Cognitive Science 14 (1990), 179–211.

[5] Hamilton, W. D. The evolution of altruistic behavior. The American Naturalist 97, 896

(1963), 354–356.

[6] Kmmerli, R., Colliard, C., Fiechter, N., Petitpierre, B., Russier, F., and Keller,

L. Human cooperation in social dilemmas: comparing the snowdrift game with the prisoner’s

dilemma. Proceedings of the Royal Society B 274 (2007), 29652970.

[7] Madsen, E. A., Tunney, R. J., Fieldman, G., Plotkin, H. C., Dunbar, R. I. M.,

Richardson, J.-M., and McFarland, D. Kinship and altruism: A cross-cultural experi-

mental study. British Journal of Psychology (2006), 1–22.

[8] Mastrianni, A., and do Nascimento Farias, L. F. Using neat to evolve cooperation and

intelligence in game theory problems. Swarthmore CS81 (Fall 2015), 1–9.

[9] McNally, L., Brown, S. P., and Jackson, A. L. Cooperation and the evolution of

intelligence. Proceedings of the Royal Society B (2012), 1–8.

[10] McNally, L., Brown, S. P., and Jackson, A. L. Cooperation and the evolution of

intelligence. Proceedings of the Royal Society B (2012), 7.

[11] McNally, L., Brown, S. P., and Jackson, A. L. Cooperation and the evolution of

intelligence. Proceedings of the Royal Society B (2012), 8.

39
[12] Mitchell, M. An Introduction to Genetic Algorithms. MIT Press, Cambridge, Massachusetts,

1996.

[13] Mitchell A. Potter, L. A. M., and Schultz, A. C. Heterogeneity in the coevolved

behaviors of mobile robots: The emergence of specialists. Proceedings of the 17th international

joint conference on Artificial intelligence 2 (2001), 1337–1343.

[14] Schultz, T. Computational Developmental Psychology. MIT Press, Cambridge, Mas-

sachusetts, 1995.

[15] Stanley, K. O., and Miikkulainen, R. Competitive coevolution through evolutionary

complexification. Journal of Artificial Intelligence Research 21 (2004), 63–100.

[16] Yong, C. H., and Miikkulainen, R. Cooperative coevolution of multi-agent systems. Tech.

Rep. AI07-338, Department of Computer Sciences, The University of Texas at Austin, 2001.

40