The Coming Technological Singularity:

How to Survive in the Post-Human Era

Vernor Vinge
Department of Mathematical Sciences
San Diego State niversity
!c" #$$% &y Vernor Vinge
!Ver&atim copying'translation an( (istri&ution of this
entire article is permitte( in any me(ium) provi(e( this
notice is preserve(*"

This article was for the V+S+,--.# Symposium
sponsore( &y -/S/ 0ewis 1esearch Center
an( the ,hio /erospace +nstitute) March %2-%#) #$$%*
+t is also retrieva&le from the -/S/ technical reports
server as part of -/S/ CP-#2#.$*
/ slightly change( version appeare( in the
3inter #$$% issue of 43hole Earth 1eview4*


/&stract
3ithin thirty years) we will have the technological
means to create superhuman intelligence* Shortly after)
the human era will &e en(e(*
+s such progress avoi(a&le5 +f not to &e avoi(e() can
events &e gui(e( so that we may survive5 These 6uestions
are investigate(* Some possi&le answers !an( some further
(angers" are presente(*
43hat is The Singularity54
The acceleration of technological progress has &een the central
feature of this century* + argue in this paper that we are on the e(ge
of change compara&le to the rise of human life on Earth* The precise
cause of this change is the imminent creation &y technology of
entities with greater than human intelligence* There are several means
&y which science may achieve this &rea7through !an( this is another
reason for having confi(ence that the event will occur":
o The (evelopment of computers that are 8awa7e8 an(
superhumanly intelligent* !To (ate) most controversy in the
area of /+ relates to whether we can create human e6uivalence
in a machine* 9ut if the answer is 8yes) we can8) then there
is little (ou&t that &eings more intelligent can &e constructe(
shortly thereafter*
o 0arge computer networ7s !an( their associate( users" may 8wa7e
up8 as a superhumanly intelligent entity*
o Computer'human interfaces may &ecome so intimate that users
may reasona&ly &e consi(ere( superhumanly intelligent*
o 9iological science may fin( ways to improve upon the natural
human intellect*
The first three possi&ilities (epen( in large part on
improvements in computer har(ware* Progress in computer har(ware has
followe( an ama:ingly stea(y curve in the last few (eca(es ;#<=* 9ase(
largely on this tren() + &elieve that the creation of greater than
human intelligence will occur (uring the ne>t thirty years* !Charles
Platt ;#$= has pointe( out the /+ enthusiasts have &een ma7ing claims
li7e this for the last thirty years* ?ust so +@m not guilty of a
relative-time am&iguity) let me more specific: +@ll &e surprise( if
this event occurs &efore .22A or after .2%2*"
3hat are the conse6uences of this event5 3hen greater-than-human
intelligence (rives progress) that progress will &e much more rapi(*
+n fact) there seems no reason why progress itself woul( not involve
the creation of still more intelligent entities -- on a still-shorter
time scale* The &est analogy that + see is with the evolutionary past:
/nimals can a(apt to pro&lems an( ma7e inventions) &ut often no faster
than natural selection can (o its wor7 -- the worl( acts as its own
simulator in the case of natural selection* 3e humans have the a&ility
to internali:e the worl( an( con(uct 8what if@s8 in our hea(sB we can
solve many pro&lems thousan(s of times faster than natural selection*
-ow) &y creating the means to e>ecute those simulations at much higher
spee(s) we are entering a regime as ra(ically (ifferent from our human
past as we humans are from the lower animals*
Crom the human point of view this change will &e a throwing away
of all the previous rules) perhaps in the &lin7 of an eye) an
e>ponential runaway &eyon( any hope of control* Developments that
&efore were thought might only happen in 8a million years8 !if ever"
will li7ely happen in the ne>t century* !+n ;D=) Ereg 9ear paints a
picture of the maFor changes happening in a matter of hours*"
+ thin7 it@s fair to call this event a singularity !8the
Singularity8 for the purposes of this paper"* +t is a point where our
mo(els must &e (iscar(e( an( a new reality rules* /s we move closer
an( closer to this point) it will loom vaster an( vaster over human
affairs till the notion &ecomes a commonplace* Get when it finally
happens it may still &e a great surprise an( a greater un7nown* +n
the #$A2s there were very few who saw it: Stan lam ;.H= paraphrase(
?ohn von -eumann as saying:
,ne conversation centere( on the ever accelerating progress of
technology an( changes in the mo(e of human life) which gives the
appearance of approaching some essential singularity in the
history of the race &eyon( which human affairs) as we 7now them)
coul( not continue*
Von -eumann even uses the term singularity) though it appears he
is still thin7ing of normal progress) not the creation of superhuman
intellect* !Cor me) the superhumanity is the essence of the
Singularity* 3ithout that we woul( get a glut of technical riches)
never properly a&sor&e( !see ;.D="*"
+n the #$<2s there was recognition of some of the implications of
superhuman intelligence* +* ?* Eoo( wrote ;#2=:
0et an ultraintelligent machine &e (efine( as a machine
that can far surpass all the intellectual activities of any
any man however clever* Since the (esign of machines is one of
these intellectual activities) an ultraintelligent machine coul(
(esign even &etter machinesB there woul( then un6uestiona&ly
&e an 8intelligence e>plosion)8 an( the intelligence of man
woul( &e left far &ehin(* Thus the first ultraintelligent
machine is the 4last4 invention that man nee( ever ma7e)
provi(e( that the machine is (ocile enough to tell us how to
7eep it un(er control*
***
+t is more pro&a&le than not that) within the twentieth century)
an ultraintelligent machine will &e &uilt an( that it will &e
the last invention that man nee( ma7e*
Eoo( has capture( the essence of the runaway) &ut (oes not pursue
its most (istur&ing conse6uences* /ny intelligent machine of the sort
he (escri&es woul( not &e human7in(@s 8tool8 -- any more than humans
are the tools of ra&&its or ro&ins or chimpan:ees*
Through the @<2s an( @H2s an( @I2s) recognition of the cataclysm
sprea( ;.I= ;#= ;%2= ;D=* Perhaps it was the science-fiction writers
who felt the first concrete impact* /fter all) the 8har(8
science-fiction writers are the ones who try to write specific stories
a&out all that technology may (o for us* More an( more) these writers
felt an opa6ue wall across the future* ,nce) they coul( put such
fantasies millions of years in the future ;.%=* -ow they saw that
their most (iligent e>trapolations resulte( in the un7nowa&le ***
soon* ,nce) galactic empires might have seeme( a Post-Human (omain*
-ow) sa(ly) even interplanetary ones are*
3hat a&out the @$2s an( the @22s an( the @#2s) as we sli(e towar(
the e(ge5 How will the approach of the Singularity sprea( across the
human worl( view5 Cor a while yet) the general critics of machine
sapience will have goo( press* /fter all) till we have har(ware as
powerful as a human &rain it is pro&a&ly foolish to thin7 we@ll &e
a&le to create human e6uivalent !or greater" intelligence* !There is
the far-fetche( possi&ility that we coul( ma7e a human e6uivalent out
of less powerful har(ware) if were willing to give up spee() if we
were willing to settle for an artificial &eing who was literally slow
;.$=* 9ut it@s much more li7ely that (evising the software will &e a
tric7y process) involving lots of false starts an( e>perimentation* +f
so) then the arrival of self-aware machines will not happen till after
the (evelopment of har(ware that is su&stantially more powerful than
humans@ natural e6uipment*"
9ut as time passes) we shoul( see more symptoms* The (ilemma felt
&y science fiction writers will &e perceive( in other creative
en(eavors* !+ have hear( thoughtful comic &oo7 writers worry a&out
how to have spectacular effects when everything visi&le can &e
pro(uce( &y the technically commonplace*" 3e will see automation
replacing higher an( higher level Fo&s* 3e have tools right now
!sym&olic math programs) ca('cam" that release us from most low-level
(ru(gery* ,r put another way: The wor7 that is truly pro(uctive is the
(omain of a stea(ily smaller an( more elite fraction of humanity* +n
the coming of the Singularity) we are seeing the pre(ictions of 4true4
technological unemployment finally come true*
/nother symptom of progress towar( the Singularity: i(eas
themselves shoul( sprea( ever faster) an( even the most ra(ical will
6uic7ly &ecome commonplace* 3hen + &egan writing) it seeme( very easy
to come up with i(eas that too7 (eca(es to percolate into the cultural
consciousnessB now the lea( time seems more li7e eighteen months* !,f
course) this coul( Fust &e me losing my imagination as + get ol() &ut
+ see the effect in others too*" 0i7e the shoc7 in a compressi&le
flow) the Singularity moves closer as we accelerate through the
critical spee(*
/n( what of the arrival of the Singularity itself5 3hat can &e
sai( of its actual appearance5 Since it involves an intellectual
runaway) it will pro&a&ly occur faster than any technical revolution
seen so far* The precipitating event will li7ely &e une>pecte( --
perhaps even to the researchers involve(* !89ut all our previous
mo(els were catatonicJ 3e were Fust twea7ing some parameters****8" +f
networ7ing is wi(esprea( enough !into u&i6uitous em&e((e( systems") it
may seem as if our artifacts as a whole ha( su((enly wa7ene(*
/n( what happens a month or two !or a (ay or two" after that5 +
have only analogies to point to: The rise of human7in(* 3e will &e in
the Post-Human era* /n( for all my rampant technological optimism)
sometimes + thin7 +@( &e more comforta&le if + were regar(ing these
transcen(ental events from one thousan( years remove *** instea( of
twenty*

4Can the Singularity &e /voi(e(54
3ell) may&e it won@t happen at all: Sometimes + try to imagine
the symptoms that we shoul( e>pect to see if the Singularity is not to
(evelop* There are the wi(ely respecte( arguments of Penrose ;#I= an(
Searle ;.#= against the practicality of machine sapience* +n /ugust
of #$$.) Thin7ing Machines Corporation hel( a wor7shop to investigate
the 6uestion 8How 3e 3ill 9uil( a Machine that Thin7s8 ;Thearling=* /s
you might guess from the wor7shop@s title) the participants were not
especially supportive of the arguments against machine intelligence*
+n fact) there was general agreement that min(s can e>ist on
non&iological su&strates an( that algorithms are of central importance
to the e>istence of min(s* However) there was much (e&ate a&out the
raw har(ware power that is present in organic &rains* / minority felt
that the largest #$$. computers were within three or(ers of magnitu(e
of the power of the human &rain* The maFority of the participants
agree( with Moravec@s estimate ;#<= that we are ten to forty years
away from har(ware parity* /n( yet there was another minority who
pointe( to ;<= ;.2=) an( conFecture( that the computational competence
of single neurons may &e far higher than generally &elieve(* +f so)
our present computer har(ware might &e as much as 4ten4 or(ers of
magnitu(e short of the e6uipment we carry aroun( in our hea(s* +f this
is true !or for that matter) if the Penrose or Searle criti6ue is
vali(") we might never see a Singularity* +nstea() in the early @22s
we woul( fin( our har(ware performance curves &egin to level off --
this cause( &y our ina&ility to automate the comple>ity of the (esign
wor7 necessary to support the har(ware tren( curves* 3e@( en( up with
some 4very4 powerful har(ware) &ut without the a&ility to push it
further* Commercial (igital signal processing might &e awesome)
giving an analog appearance even to (igital operations) &ut nothing
woul( ever 8wa7e up8 an( there woul( never &e the intellectual runaway
which is the essence of the Singularity* +t woul( li7ely &e seen as a
gol(en age *** an( it woul( also &e an en( of progress* This is very
li7e the future pre(icte( &y Eunther Stent* +n fact) on page #%H of
;.D=) Stent e>plicitly cites the (evelopment of transhuman
intelligence as a sufficient con(ition to &rea7 his proFections*
9ut if the technological Singularity can happen) it will* Even
if all the governments of the worl( were to un(erstan( the 8threat8
an( &e in (ea(ly fear of it) progress towar( the goal woul( continue*
+n fiction) there have &een stories of laws passe( for&i((ing the
construction of 8a machine in the form of the min( of man8 ;#.=* +n
fact) the competitive a(vantage -- economic) military) even artistic
-- of every a(vance in automation is so compelling that passing laws)
or having customs) that for&i( such things merely assures that someone
else will get them first*
Eric Dre>ler ;H= has provi(e( spectacular insight a&out how far
technical improvement may go* He agrees that superhuman intelligences
will &e availa&le in the near future -- an( that such entities pose a
threat to the human status 6uo* 9ut Dre>ler argues that we can em&e(
such transhuman (evices in rules or physical confinement such that
their results can &e e>amine( an( use( safely* This is +* ?* Eoo(@s
ultraintelligent machine) with a (ose of caution* + argue that
confinement is intrinsically impractical* Cor the case of physical
confinement: +magine yourself confine( to your house with only limite(
(ata access to the outsi(e) to your masters* +f those masters thought
at a rate -- say -- one million times slower than you) there is little
(ou&t that over a perio( of years !your time" you coul( come up with
8helpful a(vice8 that woul( inci(entally set you free* !+ call this
8fast thin7ing8 form of superintelligence 8wea7 superhumanity8* Such a
8wea7ly superhuman8 entity woul( pro&a&ly &urn out in a few wee7s of
outsi(e time* 8Strong superhumanity8 woul( &e more than cran7ing up
the cloc7 spee( on a human-e6uivalent min(* +t@s har( to say
precisely what 8strong superhumanity8 woul( &e li7e) &ut the
(ifference appears to &e profoun(* +magine running a (og min( at very
high spee(* 3oul( a thousan( years of (oggy living a(( up to any human
insight5 !-ow if the (og min( were cleverly rewire( an( 4then4 run at
high spee() we might see something (ifferent****" Most speculations
a&out superintelligence seem to &e &ase( on the wea7ly superhuman
mo(el* + &elieve that our &est guesses a&out the post-Singularity
worl( can &e o&taine( &y thin7ing on the nature of strong
superhumanity* + will return to this point later in the paper*"
The other approach to Dre>lerian confinement is to &uil( 4rules4
into the min( of the create( superhuman entity !/simov@s 0aws"* +
thin7 that performance rules strict enough to &e safe woul( also
pro(uce a (evice whose a&ility was clearly inferior to the unfettere(
versions !an( so human competition woul( favor the (evelopment of the
those more (angerous mo(els"* Still) the /simov (ream is a won(erful
one: +magine a willing slave) who has #222 times your capa&ilities in
every way* +magine a creature who coul( satisfy your every safe wish
!whatever that means" an( still have $$*$K of its time free for other
activities* There woul( &e a new universe we never really un(erstoo()
&ut fille( with &enevolent go(s !though one of 4my4 wishes might &e to
&ecome one of them"*
+f the Singularity can not &e prevente( or confine() Fust how &a(
coul( the Post-Human era &e5 3ell *** pretty &a(* The physical
e>tinction of the human race is one possi&ility* !,r as Eric Dre>ler
put it of nanotechnology: Eiven all that such technology can (o)
perhaps governments woul( simply (eci(e that they no longer nee(
citi:ensJ"* Get physical e>tinction may not &e the scariest
possi&ility* /gain) analogies: Thin7 of the (ifferent ways we relate
to animals* Some of the cru(e physical a&uses are implausi&le) yet****
+n a Post-Human worl( there woul( still &e plenty of niches where
human e6uivalent automation woul( &e (esira&le: em&e((e( systems in
autonomous (evices) self-aware (aemons in the lower functioning of
larger sentients* !/ strongly superhuman intelligence woul( li7ely &e
a Society of Min( ;#A= with some very competent components*" Some
of these human e6uivalents might &e use( for nothing more than (igital
signal processing* They woul( &e more li7e whales than humans* ,thers
might &e very human-li7e) yet with a one-si(e(ness) a 4(e(ication4
that woul( put them in a mental hospital in our era* Though none of
these creatures might &e flesh-an(-&loo( humans) they might &e the
closest things in the new enviroment to what we call human now* !+* ?*
Eoo( ha( something to say a&out this) though at this late (ate the
a(vice may &e moot: Eoo( ;##= propose( a 8Meta-Eol(en 1ule8)
which might &e paraphrase( as 8Treat your inferiors as you woul( &e
treate( &y your superiors*8 +t@s a won(erful) para(o>ical i(ea !an(
most of my frien(s (on@t &elieve it" since the game-theoretic payoff
is so har( to articulate* Get if we were a&le to follow it) in some
sense that might say something a&out the plausi&ility of such 7in(ness
in this universe*"
+ have argue( a&ove that we cannot prevent the Singularity)
that its coming is an inevita&le conse6uence of the humans@ natural
competitiveness an( the possi&ilities inherent in technology* /n( yet
*** we are the initiators* Even the largest avalanche is triggere( &y
small things* 3e have the free(om to esta&lish initial con(itions)
ma7e things happen in ways that are less inimical than others* ,f
course !as with starting avalanches") it may not &e clear what the
right gui(ing nu(ge really is:
4,ther Paths to the Singularity: +ntelligence /mplification4
3hen people spea7 of creating superhumanly intelligent &eings)
they are usually imagining an /+ proFect* 9ut as + note( at the
&eginning of this paper) there are other paths to superhumanity*
Computer networ7s an( human-computer interfaces seem more mun(ane than
/+) an( yet they coul( lea( to the Singularity* + call this
contrasting approach +ntelligence /mplification !+/"* +/ is something
that is procee(ing very naturally) in most cases not even recogni:e(
&y its (evelopers for what it is* 9ut every time our a&ility to access
information an( to communicate it to others is improve() in some sense
we have achieve( an increase over natural intelligence* Even now) the
team of a PhD human an( goo( computer wor7station !even an off-net
wor7stationJ" coul( pro&a&ly ma> any written intelligence test in
e>istence*
/n( it@s very li7ely that +/ is a much easier roa( to the
achievement of superhumanity than pure /+* +n humans) the har(est
(evelopment pro&lems have alrea(y &een solve(* 9uil(ing up from within
ourselves ought to &e easier than figuring out first what we really
are an( then &uil(ing machines that are all of that* /n( there is at
least conFectural prece(ent for this approach* Cairns-Smith ;A= has
speculate( that &iological life may have &egun as an a(Funct to still
more primitive life &ase( on crystalline growth* 0ynn Margulis ;#D=
has ma(e strong arguments for the view that mutualism is the great
(riving force in evolution*
-ote that + am not proposing that /+ research &e ignore( or less
fun(e(* 3hat goes on with /+ will often have applications in +/) an(
vice versa* + am suggesting that we recogni:e that in networ7 an(
interface research there is something as profoun( !an( potential wil("
as /rtificial +ntelligence* 3ith that insight) we may see proFects
that are not as (irectly applica&le as conventional interface an(
networ7 (esign wor7) &ut which serve to a(vance us towar( the
Singularity along the +/ path*
Here are some possi&le proFects that ta7e on special
significance) given the +/ point of view:
o Human'computer team automation: Ta7e pro&lems that are normally
consi(ere( for purely machine solution !li7e hill-clim&ing
pro&lems") an( (esign programs an( interfaces that ta7e a
a(vantage of humans@ intuition an( availa&le computer har(ware*
Consi(ering all the &i:arreness of higher (imensional
hill-clim&ing pro&lems !an( the neat algorithms that have &een
(evise( for their solution") there coul( &e some very interesting
(isplays an( control tools provi(e( to the human team mem&er*
o Develop human'computer sym&iosis in art: Com&ine the graphic
generation capa&ility of mo(ern machines an( the esthetic
sensi&ility of humans* ,f course) there has &een an enormous
amount of research in (esigning computer ai(s for artists) as
la&or saving tools* +@m suggesting that we e>plicitly aim for a
greater merging of competence) that we e>plicitly recogni:e the
cooperative approach that is possi&le* Larl Sims ;..= has (one
won(erful wor7 in this (irection*
o /llow human'computer teams at chess tournaments* 3e alrea(y
have programs that can play &etter than almost all humans* 9ut
how much wor7 has &een (one on how this power coul( &e use( &y a
human) to get something even &etter5 +f such teams were allowe(
in at least some chess tournaments) it coul( have the positive
effect on +/ research that allowing computers in tournaments ha(
for the correspon(ing niche in /+*
o Develop interfaces that allow computer an( networ7 access without
re6uiring the human to &e tie( to one spot) sitting in front of a
computer* !This is an aspect of +/ that fits so well with 7nown
economic a(vantages that lots of effort is alrea(y &eing spent on
it*"
o Develop more symmetrical (ecision support systems* / popular
research'pro(uct area in recent years has &een (ecision support
systems* This is a form of +/) &ut may &e too focusse( on
systems that are oracular* /s much as the program giving the user
information) there must &e the i(ea of the user giving the
program gui(ance*
o se local area nets to ma7e human teams that really wor7 !ie)
are more effective than their component mem&ers"* This is
generally the area of 8groupware8) alrea(y a very popular
commercial pursuit* The change in viewpoint here woul( &e to
regar( the group activity as a com&ination organism* +n one
sense) this suggestion might &e regar(e( as the goal of inventing
a 81ules of ,r(er8 for such com&ination operations* Cor instance)
group focus might &e more easily maintaine( than in classical
meetings* E>pertise of in(ivi(ual human mem&ers coul( &e isolate(
from ego issues such that the contri&ution of (ifferent mem&ers
is focusse( on the team proFect* /n( of course share( (ata &ases
coul( &e use( much more conveniently than in conventional
committee operations* !-ote that this suggestion is aime( at team
operations rather than political meetings* +n a political
setting) the automation (escri&e( a&ove woul( simply enforce the
power of the persons ma7ing the rulesJ"
o E>ploit the worl(wi(e +nternet as a com&ination human'machine
tool* ,f all the items on the list) progress in this is
procee(ing the fastest an( may run us into the Singularity &efore
anything else* The power an( influence of even the present-(ay
+nternet is vastly un(erestimate(* Cor instance) + thin7 our
contemporary computer systems woul( &rea7 un(er the weight of
their own comple>ity if it weren@t for the e(ge that the SE-ET
8group min(8 gives the system a(ministration an( support peopleJ"
The very anarchy of the worl(wi(e net (evelopment is evi(ence of
its potential* /s connectivity an( &an(wi(th an( archive si:e an(
computer spee( all increase) we are seeing something li7e 0ynn
Margulis@ ;#D= vision of the &iosphere as (ata processor
recapitulate() &ut at a million times greater spee( an( with
millions of humanly intelligent agents !ourselves"*
The a&ove e>amples illustrate research that can &e (one within
the conte>t of contemporary computer science (epartments* There are
other para(igms* Cor e>ample) much of the wor7 in /rtificial
+ntelligence an( neural nets woul( &enefit from a closer connection
with &iological life* +nstea( of simply trying to mo(el an( un(erstan(
&iological life with computers) research coul( &e (irecte( towar( the
creation of composite systems that rely on &iological life for
gui(ance or for the provi(ing features we (on@t un(erstan( well enough
yet to implement in har(ware* / long-time (ream of science-fiction has
&een (irect &rain to computer interfaces ;.= ;.I=* +n fact) there is
concrete wor7 that can &e (one !an( has &een (one" in this area:
o 0im& prosthetics is a topic of (irect commercial applica&ility*
-erve to silicon trans(ucers can &e ma(e ;#%=* This is an
e>citing) near-term step towar( (irect communcation*
o Similar (irect lin7s into &rains may &e feasi&le) if the &it
rate is low: given human learning fle>i&ility) the actual
&rain neuron targets might not have to &e precisely selecte(*
Even #22 &its per secon( woul( &e of great use to stro7e
victims who woul( otherwise &e confine( to menu-(riven
interfaces*
o Plugging in to the optic trun7 has the potential for &an(wi(ths
of # M&it'secon( or so* 9ut for this) we nee( to 7now the
fine-scale architecture of vision) an( we nee( to place an
enormous we& of electro(es with e>6uisite precision* +f we want
our high &an(wi(th connection to &e 4in a((ition4 to what paths
are alrea(y present in the &rain) the pro&lem &ecomes vastly more
intracta&le* ?ust stic7ing a gri( of high-&an(wi(th receivers
into a &rain certainly won@t (o it* 9ut suppose that the
high-&an(wi(th gri( were present while the &rain structure was
actually setting up) as the em&ryo (evelops* That suggests:
o /nimal em&ryo e>periments* + woul(n@t e>pect any +/ success
in the first years of such research) &ut giving (eveloping &rains
access to comple> simulate( neural structures might &e very
interesting to the people who stu(y how the em&ryonic &rain
(evelops* +n the long run) such e>periments might pro(uce
animals with a((itional sense paths an( interesting intellectual
a&ilities*

,riginally) + ha( hope( that this (iscussion of +/ woul( yiel(
some clearly safer approaches to the Singularity* !/fter all) +/
allows our participation in a 7in( of transcen(ance*" /las) loo7ing
&ac7 over these +/ proposals) a&out all + am sure of is that they
shoul( &e consi(ere() that they may give us more options* 9ut as for
safety *** well) some of the suggestions are a little scarey on their
face* ,ne of my informal reviewers pointe( out that +/ for in(ivi(ual
humans creates a rather sinister elite* 3e humans have millions of
years of evolutionary &aggage that ma7es us regar( competition in a
(ea(ly light* Much of that (ea(liness may not &e necessary in to(ay@s
worl() one where losers ta7e on the winners@ tric7s an( are coopte(
into the winners@ enterprises* / creature that was &uilt 4(e novo4
might possi&ly &e a much more &enign entity than one with a 7ernel
&ase( on fang an( talon* /n( even the egalitarian view of an +nternet
that wa7es up along with all man7in( can &e viewe( as a nightmare
;.A=*
The pro&lem is not that the Singularity represents simply the
passing of human7in( from center stange) &ut that it contra(icts some
of our most (eeply hel( notions of &eing* + thin7 a closer loo7 at the
notion of strong superhumanity can show why that is*
4Strong Superhumanity an( the 9est 3e Can /s7 for4
Suppose we coul( tailor the Singularity* Suppose we coul( attain
our most e>travagant hopes* 3hat then woul( we as7 for:
That humans themselves woul( &ecome their own successors) that
whatever inFustice occurs woul( &e tempere( &y our 7nowle(ge of our
roots* Cor those who remaine( unaltere() the goal woul( &e &enign
treatment !perhaps even giving the stay-&ehin(s the appearance of
&eing masters of go(li7e slaves"* +t coul( &e a gol(en age that also
involve( progress !overleaping Stent@s &arrier"* +mmortality !or at
least a lifetime as long as we can ma7e the universe survive ;$=
;%=" woul( &e achieva&le*
9ut in this &rightest an( 7in(est worl() the philosophical
pro&lems themselves &ecome intimi(ating* / min( that stays at the same
capacity cannot live foreverB after a few thousan( years it woul( loo7
more li7e a repeating tape loop than a person* !The most chilling
picture + have seen of this is in ;#H=*" To live in(efinitely long)
the min( itself must grow *** an( when it &ecomes great enough) an(
loo7s &ac7 *** what fellow-feeling can it have with the soul that it
was originally5 Certainly the later &eing woul( &e everything the
original was) &ut so much vastly more* /n( so even for the in(ivi(ual)
the Cairns-Smith !or 0ynn Margulis" notion of new life growing
incrementally out of the ol( must still &e vali(*
This 8pro&lem8 a&out immortality comes up in much more (irect
ways* The notion of ego an( self-awareness has &een the &e(roc7 of
the har(hea(e( rationalism of the last few centuries* Get now the
notion of self-awareness is un(er attac7 from the /rtificial
+ntelligence people !8self-awareness an( other (elusions8"*
+ntelligence /mplification un(ercuts the importance of ego from
another (irection* The post-Singularity worl( will involve e>tremely
high-&an(wi(th networ7ing* / central feature of strongly superhuman
entities will li7ely &e their a&ility to communicate at varia&le
&an(wi(ths) inclu(ing ones far higher than speech or written messages*
3hat happens when pieces of ego can &e copie( an( merge() when the
si:e of a selfawareness can grow or shrin7 to fit the nature of the
pro&lems un(er consi(eration5 These are essential features of strong
superhumanity an( the Singularity* Thin7ing a&out them) one &egins to
feel how essentially strange an( (ifferent the Post-Human era will &e
-- 4no matter how cleverly an( &enignly it is &rought to &e4*

Crom one angle) the vision fits many of our happiest (reams:
a place unen(ing) where we can truly 7now one another an( un(erstan(
the (eepest mysteries* Crom another angle) it@s a lot li7e the worst
case scenario + imagine( earlier in this paper*
3hich is the vali( viewpoint5 +n fact) + thin7 the new era is
simply too (ifferent to fit into the classical frame of goo( an(
evil* That frame is &ase( on the i(ea of isolate() immuta&le min(s
connecte( &y tenuous) low-&an(with lin7s* 9ut the post-Singularity
worl( 4(oes4 fit with the larger tra(ition of change an( cooperation
that starte( long ago !perhaps even &efore the rise of &iological
life"* + thin7 there 4are4 notions of ethics that woul( apply in such
an era* 1esearch into +/ an( high-&an(wi(th communications shoul(
improve this un(erstan(ing* + see Fust the glimmerings of this now)
in Eoo(@s Meta-Eol(en 1ule) perhaps in rules for (istinguishing self
from others on the &asis of &an(wi(th of connection* /n( while min(
an( self will &e vastly more la&ile than in the past) much of what we
value !7nowle(ge) memory) thought" nee( never &e lost* + thin7
Creeman Dyson has it right when he says ;I=: 8Eo( is what min( &ecomes
when it has passe( &eyon( the scale of our comprehension*8
;+ wish to than7 ?ohn Carroll of San Diego State niversity an( Howar(
Davi(son of Sun Microsystems for (iscussing the (raft version of this
paper with me*=

4/nnotate( Sources ;an( an occasional plea for &i&liographical help=4
;#= /lfvMn) Hannes) writing as ,lof ?ohanneson) 4The En( of Man54)
/war( 9oo7s) #$<$ earlier pu&lishe( as 8The Tale of the 9ig
Computer8) Cowar(-McCann) translate( from a &oo7 copyright #$<<
/l&ert 9onniers Corlag /9 with English translation copyright #$<<
&y Victor Eollan:) 0t(*
;.= /n(erson) Poul) 8Lings 3ho Die8) 4+f4) March #$<.) pI-%<*
1eprinte( in 4Seven Con6uests4) Poul /n(erson) MacMillan Co*) #$<$*
;%= 9arrow) ?ohn D* an( Cran7 ?* Tipler) 4The /nthropic Cosmological
Principle4) ,>for( niversity Press) #$I<*
;D= 9ear) Ereg) 89loo( Music8) 4/nalog Science Ciction-Science Cact4)
?une) #$I%* E>pan(e( into the novel 49loo( Music4) Morrow) #$IA
;A= Cairns-Smith) /* E*) 4Seven Clues to the ,rigin of 0ife4) Cam&ri(ge
niversity Press) #$IA*
;<= Conra() Michael 4et al*4) 8Towar(s an /rtificial 9rain8)
49ioSystems4) vol.%) pp#HA-.#I) #$I$*
;H= Dre>ler) L* Eric) 4Engines of Creation4) /nchor Press'Dou&le(ay) #$I<*
;I= Dyson) Creeman) 4+nfinite in /ll Directions4) Harper NN 1ow) #$II*
;$= Dyson) Creeman) 8Physics an( 9iology in an ,pen niverse8) 41eview
of Mo(ern Physics4) vol A#) ppDDH-D<2) #$H$*
;#2= Eoo() +* ?*) 8Speculations Concerning the Cirst ltraintelligent
Machine8) in 4/(vances in Computers4) vol <) Cran: 0* /lt an(
Morris 1u&inoff) e(s) pp%#-II) #$<A) /ca(emic Press*
;##= Eoo() +* ?*) ;HelpJ + can@t fin( the source of Eoo(@s Meta-Eol(en
1ule) though + have the clear recollection of hearing a&out it
sometime in the #$<2s* Through the help of the net) + have foun(
pointers to a num&er of relate( items* E* Harry Stine an( /n(rew
Haley have written a&out metalaw as it might relate to
e>traterrestrials: E* Harry Stine) 8How to Eet along with
E>traterrestrials *** or Gour -eigh&or8) 4/nalog Science Cact-
Science Ciction4) Ce&ruary) #$I2) p%$-DH*=

;#.= Her&ert) Cran7) 4Dune4) 9er7ley 9oo7s) #$IA* However) this novel was
seriali:e( in 4/nalog Science Ciction-Science Cact4 in the #$<2s*
;#%= Lovacs) E* T* /* 4et al*4) 81egeneration Microelectro(e /rray for
Peripheral -erve 1ecor(ing an( Stimulation8) 4+EEE Transactions
on 9iome(ical Engineering4) v %$) n $) pp I$%-$2.*
;#D= Margulis) 0ynn an( Dorion Sagan) 4Microcosmos) Cour 9illion Gears of
Evolution from ,ur Micro&ial /ncestors4) Summit 9oo7s) #$I<*
;#A= Mins7y) Marvin) 4Society of Min(4) Simon an( Schuster) #$IA*
;#<= Moravec) Hans) 4Min( Chil(ren4) Harvar( niversity Press) #$II*
;#H= -iven) 0arry) 8The Ethics of Ma(ness8) 4+f4) /pril #$<H) ppI.-#2I*
1eprinte( in 4-eutron Star4) 0arry -iven) 9allantine 9oo7s) #$<I*
;#I= Penrose) 1*) 4The Emperor@s -ew Min(4) ,>for( niversity Press) #$I$*
;#$= Platt) Charles) Private Communication*
;.2= 1asmussen) S* 4et al*4) 8Computational Connectionism within -eurons:
a Mo(el of Cytos7eletal /utomata Su&serving -eural -etwor7s8) in
4Emergent Computation4) Stephanie Correst) e(*) pD.I-DD$) M+T
Press) #$$#*
;.#= Searle) ?ohn 1*) 8Min(s) 9rains) an( Programs8) in 4The 9ehavioral an(
9rain Sciences4) v*%) Cam&ri(ge niversity Press) #$I2* The
essay is reprinte( in 4The Min(@s +4) e(ite( &y Douglas 1*
Hofsta(ter an( Daniel C* Dennett) 9asic 9oo7s) #$I#* This
reprinting contains an e>cellent criti6ue of the Searle essay*
;..= Sims) Larl) 8+nteractive Evolution of Dynamical Systems8) Thin7ing
Machines Corporation) Technical 1eport Series !pu&lishe( in 4Towar(
a Practice of /utonomous Systems: Procee(ings of the Cirst European
Cnference on /rtificial 0ife4) Paris) M+T Press) Decem&er #$$#*
;.%= Staple(on) ,laf) 4The Starma7er4) 9er7ley 9oo7s) #$<# !&ut from
the forwar( pro&a&ly written &efore #$%H"*
;.D= Stent) Eunther S*) 4The Coming of the Eol(en /ge: / View of the En(
of Progress4) The -atural History Press) #$<$*
;.A= Swanwic7 Michael) 4Vacuum Clowers4) seriali:e( in 4+saac /simov@s
Science Ciction Maga:ine4) Decem&er!5" #$I< - Ce&ruary #$IH*
1epu&lishe( &y /ce 9oo7s) #$II*
;.<= Thearling) Lurt) 8How 3e 3ill 9uil( a Machine that Thin7s8) a wor7shop
at Thin7ing Machines Corporation* Personal Communication*
;.H= lam) S*) Tri&ute to ?ohn von -eumann) 49ulletin of the /merican
Mathematical Society4) vol <D) nr %) part .) May) #$AI) p#-D$*
;.I= Vinge) Vernor) 89oo7worm) 1unJ8) 4/nalog4) March #$<<) ppI-D2*
1eprinte( in 4True -ames an( ,ther Dangers4) Vernor Vinge) 9aen
9oo7s) #$IH*
;.$= Vinge) Vernor) 8True -ames8) 49inary Star -um&er A4) Dell) #$I#*
1eprinte( in 4True -ames an( ,ther Dangers4) Vernor Vinge) 9aen
9oo7s) #$IH*
;%2= Vinge) Vernor) Cirst 3or() 4,mni4) ?anuary #$I%) p#2*