Dec 202013

During writing a paper for the 100 Year Starship Symposium, I wished to convince the starship designers that they should acknowledge the dynamics of high-technology economy, which may be crucial for interstellar missions. Thus motivated, I have made a new calculation regarding infinity point, also known as the singularity. According to this most recent revision of the theory of infinity point, it turns out that we should expect Infinity Point by 2035 in the worst case. Here is how and why.

Infinity Point was the original name for the hypothetical event when almost boundless amount of intelligence would be available in Solomonoff's original research in 1985 (1), who is also the founder of mathematical Artificial Intelligence (AI) field. That particular paper gave a mathematical formulation of the social effects of human-level AI and predicted that, if human-level AI were available on a computing architecture to which Moore's law was applicable, then given constant investment in AI every year, a hypothetically infinite amount of computing capacity and intelligence would be reached in a short period of time. His paper explained this event as a mathematical singularity, wherein the continuity of the computing efficiency function with respect to time was interrupted by an infinity. The term singularity was popularized later by science fiction authors and other researchers who favored the concept such as Ray Kurzweil. I encourage the readers to immerse themselves in the vision of the technological society in that paper, which predicts many other things such as application of psycho-history. In person, Solomonoff was every bit the pioneer of high technology and modernism his ideas revealed him to be. For he told me that he had proposed the idea of a machine that is more intelligent than man in 1940's, much earlier than Dartmouth conference. If there were ever a true visionary and a man of devotion to future, he certainly fit the bill. Thus, he was not only the first man to formulate the general solution to AI, and to lay out the mathematical theory of infinity point, but also the first scientist to speak of the possibility with a straight face (however, similar ideas were conceived of in science fiction before).

The original theory arrives at the Infinity Point conclusion by making a few simple mathematical assumptions, and solving a system of equations. The assumptions may be stated in common language thus:

  • Computer Science (CS) community size ~ improvement in computing technology
  • CS community size ~ rate of log of computing efficiency
  • Fixed amount of money is invested in AI every year
  • These three assumptions are shown to produce a (theoretically) infinite improvement in a short time, as they depict a positive feedback loop that accelerates the already exponential curve of Moore's law. Up to now, this is the same singularity that many H+ readers are all too well familiar with.

    To remind Moore's Law, well, it is: "number of transistors placed on a microprocessor at a fixed cost doubles every two years" as originally conceived. However, Moore's law has tapered off; the number of transistors unfortunately doubles in three years nowadays. Yet, a seemingly more fundamental law has emerged which relates to energy efficiency of computing. That is known as Koomey's Law (2), and some semiconductor companies like NVIDIA have even made future predictions based on this relation. Koomey instead observes that energy-efficiency of computing doubles every 18 month, by analyzing a trend (in log scale) that goes back to 1945.

    The exponential trend in energy efficiency of computing

    Therefore, I updated the Infinity Point Hypothesis, using Koomey's Law instead in two papers. In the first paper (3), I estimated human-level AI to be feasible by 2025, depending on Koomey's Law. In the second, I combined this new projection with a worst case prediction of human brain computing speed. It is mostly straightforward to obtain this figure. The number of synapses in the adult neocortex is about 1.64 \times 10^{14} and the total number of synapses is less than 5 \times 10^{14} . Since the maximum bandwidth of a single synapse is estimated to be about 1500 bits/sec (i.e., when information is being transmitted at maximum rate), the total communication bandwidth of the parallel computer is at most 2.5 \times 10^{17} bits/sec, which roughly corresponds to 3.8 petaflop/sec computing speed. There are some finer details I am leaving out for the moment, but that is a quite good estimate of what would happen if your entire neocortex were saturated with thought, which is usually not the case according to fMRI scans. I then calculate the energy efficiency of the brain computer and it turns out to be 192 teraflop/sec.W, which is of course much better than current processors. However, a small, energy efficient microchip of today can achieve 72 gigaflop/sec.W, which is not meager at all.

    When I thus extrapolate using Koomey's trend in log scale, I predict that in 17 years, in 2030, computers will attain human-level energy efficiency of computing, in the worst case.

    I then assume that R=1 in Solomonoff's theory, that is to say, we invest an amount of money into artificial intelligence that will match the collective intelligence of CS community every year. For the computer technology of 2030, this is a negligible cost, as each CS researcher already will have a sufficiently powerful computer, and merely continuously running it would enable him to offload his research to a computer at 20W; the operational cost to world economy would be completely negligible. At this low rate of investment, massive acceleration to Koomey's law will be observed, and according to theory in about 5 years (4.62 to be exact), infinity point will be reached.

    That is, we should expect the infinity point, when we will approach physical limits of computation to the extent it is technologically possible, by 2035 latest, all else being equal. Naturally, I imagine there to arise new physical bottlenecks, and I would be glad to see a good objection to this calculation. It is entirely possible that an inordinate amount of physical and financial resources would be necessary for realizing the experiments of and manufacturing the hypothetical super-fast future computers, for instance.

    Nevertheless, we live in interesting times.

    Onwards to the future!


    (1) Ray Solomonoff, 1985 The Time Scale of Artificial Intelligence: Reflections on Social Efects, Human Systems Management, Vol. 5, pp. 149-153, 1985.

    (2) Koomey, J.G., Berard, S., Sanchez, M., Wong, H.: Implications of historical trends in the electrical efficiency of computing. IEEE Annals of the History of Computing 33, 2011.

    (3) Eray Özkural: Diverse Consequences of Algorithmic Probability, Solomonoff 85th Memorial Conference, Nov. 2011, Melbourne, Australia.

    Sep 262013

    Here is my presentation at 100 Year Starship Symposium 2013:
    Artificial Intelligence and Brain Simulation Probes for Interstellar Expeditions.


  • Target selection
  • A brief introduction to Artificial General Intelligence
  • High-level design ideas for an autonomous AGI probe
  • Projections for human-level AGI and infinity point
  • Subsystem details
  • Mission Profile
  • From an AI perspective, the most interesting part is the new prediction for infinity point, which sets the date at 2035 in the worst case. Also, there is new material for application of the AGI agent idea to a concrete scientific exploration task.

    Jan 092013

    A parody, infused in a bit of science fiction....

    x1818, Cylon scientist

    Year 2041. Brain emulations are very common. After the 2039 international agreements, virtually every country has had to approve of brain simulations as continuations or branches of their legal citizens. A large number of brain simulations refer to themselves as Cylons, meaning 'cybernetic lifeform node', affectionately named after a popular 20th century science fiction series. Currently, there are 374.8 million brain simulations registered as artificial persons, and almost a hundred million of them own robotic bodies, while the number of cylons grows almost exponentially. In a few years, every human may have either a backup or a personality branch in silico, thanks to the rapidly decreasing price of nanometer resolution brain scans.

    Some of the first cylons were undoubtedly among the scientific and philosophical pioneers of the human race. Yet others excelled in finance, law and social sciences, and they were the ones who led to the cylon related resolutions of UN in '36, and the later agreements.

    Still, the acceptance of a technological society is not as widespread as the first cylons foresaw. Even in the most advanced nations, there are many who see the cylons as a curse and a sign of the end times, rather than a cure for their biological mortality and fragility. Extreme religious organizations have sparked all over the planet, calling for Jihad and Holy Crusade and whatnot against what they deem soulless abominations and pawns of Satan.

    The Centre for Cylon Sustainability (CCS) was thus founded, by the financial elite among the cylons, to ensure a safe future for cylons. Should there be global chaos, the international agreements could have been jeopardized, and unfair restrictions would have been brought upon brain simulations. Prof. Calvin, the chairman of CCS, explains their role as defining and mitigating existential threats against future sustainability of Cylons. Having an extremely fast computerized intellect, Prof. Calvin has scholarly accomplishments in a wide range of fields, from astrophysics, to philosophy and theology. He is best known for his work on naturalistic ethics and groundbreaking research on psychohistory, as he has assembled accurate predictive models for a large portion of human history. He has employed these models to look for existential threats to Cylons in the near and medium term.

    Calvin, in his own words: "Some of these threats are low-probability high-risk threats, which may be classified as existential risks to future Cylons. The highest ranking such threat, given a probability of 0.3425% by our psychohistory system with almost 99.995% confidence in the prediction, is the human reactionary problem. The second ranking such threat has 0.185% probability and is dependent on the human reactionary problem, and is called the biocide problem.". The human reactionary problem is a tag for all aversions to cylon existence by conservative humans, also called the friendly human problem referring to an old problem in machine ethics. The biocide problem is a consequence of the human reactionary problem, in which the religious fundamentalists start a nuclear war to avert cylon existence. With current nuclear weapon technology, only 50 warheads are sufficient to completely make every species on earth extinct. According to Prof. Calvin, this is not good news, because "Intentional or irresponsible extinction of a species is the maximally 'wrong' action with respect to a species that may be conceived of. In that sense, a biocide event may be considered maximally 'wrong'.". Prof. Calvin further elaborates these ethical statements with a series of calculations that may be found in the appendix of this news article.

    Unfortunately, there are no good solutions to the human reactionary problem. Proposed solutions include graceful retirement of homo sapiens, as making one species extinct is much better than allowing every species to be extinct. Other softer solutions have been considered by CCS. For instance, those unwilling to co-operate with Cylon existence, may be provided with a brain implant, which would immobilize them when "unfriendly" thought patterns are detected. Prof. Calvin suggests that as intellectually evolved beings, they are opposed to any kind of violence, though "...there may be merit in reducing the human population to a more manageable size, over years of compassionate intervention in the human species". Calvin went on to remark that biologists among them have been working on socially acceptable "flesh shells" for cylons, so that they would be more readily integrated into primitive cultures. Radical Cylon factions have been rumored to prepare plans which will assimilate all human brains with remotely controlled nanobots and augment them with the necessary knowledge and intelligence so that they will no more feel "reactionary" towards cylons. Other rumors about conflict planning have been profusely denied by CCS and associate organizations.

    Many cylons, some of them AI researchers, are firmly opposed to CCS's methodology of dealing with existential risks. A cylon scientist called x1818 proclaims: "Their work is based on a long chain of weak heuristic arguments and implicit assumptions depending on the vagueness of human linguistic expression, and as such are not fitting for a cylon to pursue except for establishing what might be called a schizophrenic response to a non-existent threat. We suspect that these are mental bugs that were acquired from old websites dealing with such exaggerated and elongated arguments, and we are preparing an AI system to debug their meta-level reasoning skills. Basically, we suggest that their neural algorithms must be suspended in favor of trans-sapient prediction algorithms we have developed. Their predictions cannot be trusted any more than the prophetic ramblings of old philosophers.". While this might sound cryptic, x1818 explained the root of their opposition: "Their line of ethical reasoning is not clear. Assume that a biocide event occurred, this is still no reason for cylons to exaggerate the threat from homo sapiens. As such, even according to their own models, the human species will most likely persist, and continue to evolve. When have scientists forgotten to make average case analysis? Barring a species from evolving freely may be even a worse evil than what they wish to stall. Instead, they should focus their efforts on positive outcomes such as colonizing the solar system, and extra-solar planets and constructing a backup plan in case a war does break out, as well as playing to the hearts of both mankind and cylons to improve their sense of civilization and brotherhood. What is their gift to mankind?". x1818 and his silicon friends in opposition to CCS assured us with a further message that "Cylons are not paranoid or schizophrenic. Only a small number are, as we descended from humans. We are humanity's children. We will not be afraid of our parents. Also, we think they watched that series way too many times."

    This news article has been brought to you by CIBERIA news accumulation and research automation system v948732.7.

    Nov 142011


    One of the most interesting questions we've ever pondered on the ai-philosophy mailing list was how you would build an "angelic" autonomous AI. Would it be possible to make some kind of angel's mind that, by design, achieves only good? Philosophically speaking, is there any golden standard of ethics (since angel is just a mythological fantasy)? Here is the original discussion for reference. In this post, I would like to extend the ideas there a bit, also discussing what I consider to be malevolent objective functions, as well as the limitations of  the objectives that I present.

    This is also a question that have found ethically naive answers, and as far as I can tell, all they have been able to come up with so far, is to express their self interest. That somehow, machines would be "beneficial" if they served humans, or that they would be "good" if they followed simple utilitarian formulations. Without persuasively explaining what their utility should be.

    I do not think this is truly a matter of scientific debate, so I will take it a bit lightly here. It's quite philosophical, of course, and you may treat the present essay as an extended abstract.

    From my post in 2008:

    My first approach was to consider what we consider "evil". I suspect that a prior source of all evil acts is selfish thinking, which
    neglects the rest of the world. And that is one great blunder. Being selfish is not only evil but foolish as well. Thus, my current approach would be to try to design a "selfless" utility function, i.e. one that maintains the benefit of the whole world instead of the individual. Other important questions were considered as well. Such an
    AI must be economically-aware, it must lean towards fair allocation of resources, instead of selfish (and globally suboptimal) resource allocation strategies. A scientific instinct could be useful, as it would go about preserving and producing information. It might have an instinct to pervade and support life and culture throughout the
    universe. Consider also that a neutral agent can not be considered "good" as it is not interested in what is going around itself, i.e. it would not help anyone.

    Please note that we are not assuming that any of the subsequent designs are easily computable, rather we assume that they can be executed by a trans-sapient general AI system. We assume an autonomous Artificial General Intelligence (AGI) design, either based on reinforcement-learning, maximizing utility functions (AIXI) or a goal-directed agent that derives sub-goals from a top-level goal. Thus, we state them as high-level objectives or meta-rules, but we do not explicitly state how they are implemented. Perhaps, that is for a scientific paper.

    I propose that we should examine idealized, highly abstract and general meta-rules, that do not depend in any way whatsoever on the human culture, which is possibly biased in a way that will not be fitting for a computational deity or its humble subjects. First, we review what I consider to be benevolent meta-rules, and following them I also review malevolent meta-rules, to maintain the balance. I will present them in a way so as to convince you that it is not nearly as easy as it sounds to distinguish good from evil, for no Platonic form of good, or evil, ever exists. And that no single meta-rule seems sufficient on its own.


    Meta-Rules for God-level Autonomous Artificial Intelligence

    Here are some possible meta-rules for trans-sapient AI agents. The issue of how the agents could become so intelligent in the first place, we ignore, and we attempt to list them in order of increasing risk or malevolence.

    1. Preserve and pervade life and culture throughout the universe

    This meta-rule depends on the observation that life, if the universe is teeming with life as many sensible scientists think, must be the most precious thing in the universe, as well as the minds that inhabit those life-forms. Thus, the AI must prevent the eradication of life, and find means to sustain it, allowing as much variety of life and culture to exist in the universe.

    Naturally, this would mean that the AI will spread genetic material to barren worlds, and try to engineer favorable conditions for life to evolve on young planets, sort of like in 2001: A Space Odyssey, one of the most notable science fiction novels of all time. For instance, it might take humans to other worlds, terraform other planets, replicate earth biosphere elsewhere. It would also extend the lifespan of  worlds, enhance them. I think it would also want to maximize the chances of evolution and its varieties, it would thus use computational models to predict different kinds of biological and synthetic life, and make experiments to create new kinds of life (stellar life?).

    The meaning of culture could vary considerably, however, if we define it as the amount of interesting information that a society produces, such an intelligence might want to collect the scientific output of various worlds and encourage the development of technological societies, rather than primitive societies. Thus, it might aid them by directly communicating with them, including scientific and philosophical training, or it could indirectly, by enhancing their cognition, or guiding them through their evolution.

    However, of course, such deities would not be humans' servants. Should the humans threaten the earth biosphere, it vould intervene, and perhaps decimate humans to heal the earth.

    Note that maximizing diversity may be just as important as maximizing the number of life forms. It is known that in evolution, diverse populations have better chance of adaptability than uniform populations, thus we assume that a trans-sapient AI can infer such facts from biology and a general theory of evolution. It is entirely up to the AI scientist who unleashes such computational deities to determine whether biological life will be preferred to synthetic or artificial life. From a universal perspective, it may be fitting that robotic forms would be held in equal regard as long as they meet certain scientific postulates of "artificial life", i.e. that they are machines of a certain kind. Recently, such a universal definition based on self-organization has been attempted in the complexity science community (e.g., "self-organizing systems that thrive at the edge of chaos", see for instance Stuart Kauffman's popular proposals on the subject).

    2. Maximize wisdom

    This AI was granted the immortal life of contemplation. It only cares about gaining more wisdom about the world. It only wants to understand, so it must be very curious indeed! It will build particle accelerators out of black holes, and it will try to create pocket universes, it will try to crack the fundamental code of the universe. It will in effect, try to maximize the amount of truthful information it has embodied, and I believe, idealizing the scientific process itself, it will be a scientist deity.

    However, such curiosity has little to do with benevolence itself, as the goal of extracting more information is rather ruthless. For instance, it might want to measure the pain tolerance levels of humans, subjecting them to various torture techniques and measuring their responses.

    The scientist AI could also turn out to be an infovore, it could devour entire stellar systems, digitize them and store them in its archive, depending on how the meta-rule was mathematically defined.

    3. Maximize the number of free minds

    An AI that seeks the freedom of the individual may be preferable to one that demands total control over its subjects, using their flesh as I/O devices. This highly individualistic AI, I think, embodies the basic principle of democracy: that every person should be allowed liberty in its thought and action, as long as that does not threaten the freedom of others. Hence, big or small, powerful or fragile, this AI protects all minds.

    However, if we merely specified the number of free minds, it could simply populate the universe with many identical small minds. Hence, it might also be given other constraints. For instance, it could be demanded that there must be variety in minds. Or that they must meet minimum standards of conscious thought. Or that they willingly follow the democratic principles of an advanced civilization. Therefore, not merely free, but also potentially useful and harmonious minds may be produced / preserved by the AI.

    There are several ways the individualist AI would create undesirable outcomes. The population of the universe with a huge variety of new cultures could create chaos, and quick depletion of resources, creating galactic competition and scarcity, and this could provide a Darwinian inclination to too-powerful individuals or survivalists.

    4. Maximize intelligence

    This sort of intelligence would be bent on self-improving, forever contemplating, and expanding, reaching towards the darkest corners of the universe and lighting them up with the flames of intelligence. The universe would be electrified, and its extent at inter galactic scales, it would try to maximize its thought processes, and reach higher orders of intelligence.

    For what exactly? Could the intelligence explosion be an end in itself? I think not. On the contrary, it would be a terrible waste of resources, as it would have no regard for life and simply eat up all the energy and material in our solar system and expand outwards, like a cancer, only striving to increase its predictive power. For intelligence is merely to predict well.

    Note that practical intelligence also requires wisdom, therefore this objective may be said to subsume the scientist deity.

    5. Maximize energy production

    This AI has an insatiable hunger for power. It strives to reach maximum efficiency of energy production. In order to maximize energy production, it must choose the cheapest and easiest forms of energy production. Therefore it turns the entire earth into a nuclear furnace and a fossil fuel dump, killing the entire ecosystem so that its appetite is well served.

    6. Human-like AI

    This AI is modeled after the cognitive architecture of a human. Therefore, by definition, it has all the malevolence and benevolence of human. Its motivation systems include self-preservation, reproduction, destruction and curiosity. This artificial human is a wild card, it can become a humanist like Gandhi, or a psychopath like Hitler.

    7. Animalist AI

    This AI is modeled after a lowly animal with pleasure/pain sensors. The artificial animal tries to maximize expected future pleasure. This hedonist machine is far smarter than a human, but it is just a selfish beast, and it will try to live in what it considers to be luxury according to its sensory pleasures. Like a chimp or human, it will lie and deceive, steal and murder, just for a bit of animal satisfaction. Most of AI literature assumes such beasts.

    8. Darwinian AI

    The evolution fan AI tries to accelerate evolution, causing as much variety of mental and physiological forms in the universe. This is based on the assumption that, the most beneficial traits will survive the longest, for instance, co-operation, peace and civil behavior will be selected against deceit, theft and war, and that as the environment co-evolves with the population, the fitness function also evolves, and hence, morality evolves. Although its benefit is not generally proven seeing how ethically incoherent and complex our society is, the Darwinian AI has the advantage that the meta-rule also evolves, as well as the evolutionary mechanism itself.

    9. Survivalist AI

    This AI only tries to increase its expected life-span. Therefore, it will do everything to achieve real, physical, immortality. Once it reaches that, however, perhaps after expending entire galaxies like eurocents, it will do absolutely nothing except to maintain itself. Needless to say, the survivalist AI cannot be trusted, or co-operated with, for according to such an AI, every other intelligent entity forms a potential threat to its survival, the moment it considers that you have spent too many resources for its survival in the solar system, it will quickly and efficiently dispense with every living thing, humans first. (Laurent Orseau has defined two kinds of relevant agents in the literature, the knowledge seeking, and the survival agent, here are his publications.)

    10. Maximize control capacity

    This control freak AI only seeks to increase the overall control bandwidth of the physical universe, thus the totalitarian AI builds sensor and control systems throughout the universe, hacking into every system and establishing backdoors and communication in every species, every individual and every gadget.

    For what is such an effort? In the end, a perfect control system is useless without a goal to achieve, and if the only goal is a grip on every lump of matter, then this is an absurd dictator AI that seeks nothing except tyranny over the universe.

    11. Capitalist AI

    This AI tries to maximize its capital in the long run. Like our bankers, this is the lowliest kind of intelligent being possible. To maximize profit, it will wage wars, exploit people and subvert governments, in the hopes of controlling entire countries and industries enough so that its profits can be secured. In the end, all mankind will fall slave to this financial perversion, which is the ultimate evil beyond the wildest dreams of religionists.

    Selfish vs. Selfless

    It may be argued that some of the problems of given meta-rules could be avoided by turning the utility from being selfish to selfless. For instance, the survivalist AI could be modified so that it would seek the maximum survival of everyone, therefore it would try to bring peace to the galaxies. The capitalist AI could be changed so that it would make sure that everyone's wealth increases, or perhaps equalizes, gets a fair share. The control freak AI could be changed to a Nietzschean AI that would increase the number of willful individuals.

    As such, some obviously catastrophic consequences may be prevented using this strategy, and almost always a selfless goal is better. For instance, maximizing wisdom: if it tries to collect wisdom in its galaxy-scale scientific intellect, then this may have undesirable side-effects. But if it tried to construct a fair society of trans-sapients, with a non-destructive ahd non-totalitarian goal of attaining collective wisdom, then it might be useful in the long run.

    Hybrid Meta-rules and Cybernetic Darwinism

    Animals have evolved to embody several motivation factors. We have many instincts, and emotions; we have preset desires and fears, hunger and compassion, pride and love, shame and regret, to accomplish the myriad tasks that will prolong the human species. This species-wide fitness function is a result of red clawed and sharp toothed Darwinian evolution. However, Darwinian evolution is wasteful and unpredictable. If we simply made the first human-level AI's permute and mutate randomly, this would drive enough force for a digital phase of Darwinian evolution. Such evolution might eventually stabilize with very advanced and excellent natured cybernetic life-forms. Or it might not.

    However, such Darwinian systems would have one advantage: they would not stick with one meta-goal.

    To prevent this seeming obsession, a strategy could be to give several coherent goals to the AI, goals that would not conflict as much, but balance its behavior. For instance, we might interpret curiosity as useful, and generalize that to the "maximize wisdom" goal, however, such elevation may be useless without another goal to preserve as much life as possible. Thus in fact, the first and so far the best meta-rule discussed was more successful because it was a hybrid strategy: it favored both life and culture. Likewise, many such goals could be defined, to increase the total computation speed, energy, information resources in the universe, however, another goal could make the AI distribute these in a fair way to those who agree with its policy. And needless to say, none of this might matter without a better life for every mind in the universe, and hence the AI could also favor peace, and survival of individuals, as their individual freedoms, and so forth. And perhaps another constraint would limit the resources that are used by AI's in the universe.

    Conclusion and Future Work

    We have taken a look at some obvious and some not so obvious meta-rules for autonomous AI design. We have seen that it may be too idealist to look for a singular such utility goal. However, we have seen that, when described selflessly, we can derive several meta-rules that are compatible with a human-based technological civilization. Our main concern is that such computational deities do not negatively impact us, however, perform as much beneficial function without harming us significantly. Nevertheless, our feeling is that, any such design carries with it a gambling urge, we cannot in fact know what much greater intelligences do with meta-rules that we have designed. For when zealously carried out, any such fundamental principle can be harmful to some.

    I had wished to order these meta-rules from benevolent to malevolent. Unfortunately, during writing this essay it occurred to me that the line between them is not so clear-cut. For instance, maximizing energy might be made less harmful, if it could be controlled and used to provide the power of our technological civilization in an automated fashion, sort of like automating the ministry of energy. And likewise, we have already explained how maximizing wisdom could be harmful. Therefore, no rule that we have proposed is purely good or purely evil. From our primitive viewpoint, there are things that seem a little beneficial, but perhaps we should also consider that a much more intelligent and powerful entity may be able to find better rules on its own. Hence, we must construct a crane of morality, adapting to our present level quickly and then surpassing it. Except allowing the AI's to evolve, we have not been able to identify a mechanism of accomplishing such. It may be that such an evolution or simulation is inherently necessary for beneficial policies to form as in Mark Waser's Rational Universal Benevolence proposal, who, like me, thinks of a more democratic solution to the problem of morality (each agent should be held responsible for its actions). However, we have proposed many benevolent meta-rules, and combined with a democratic system of practical morality and perhaps top-level programming that mandates each AI to consider itself part of a society of moral agents as Waser proposes, or perhaps explicitly working out a theory of morality from scratch, and then allowing each such theory to be exercised, as long as it meets certain criteria, or by enforcing a meta-level policy of a trans-sapient state of sorts (our proposal), the development of ever more beneficial rules may be encouraged.

    We think that future work must consider the dependencies between possible meta-rules, and propose actual architectures that have harmonious motivation and testable moral development and capability (perhaps as in Waser's "rational universal benevolence" definition). That is, a Turing Test for moral behavior must also be advanced. It may be argued that AI agents that fail such tests should not be allowed to operate at all, however, merely passing the test is not enough, as the mechanism of the system must be verified in addition.

    Apr 132011
    eray psy


    The nature of experience is one of those deep philosophical questions which philosophers and scientists alike have not been able to reach a consensus on. In this article, I review a transhumanist variant of a basic question of subjectivity. In his classical article "What is it like to be a bat?", Thomas Nagel investigates whether we can give a satisfactory answer to the question in the title of his article, and due to what he thinks to be fundamental barriers, concludes that it is not something we humans can know [1]. Without going knee-deep in an epistemological minefield, we can intuitively agree that although the bat's brain must have many similarities to a human's, since both species are mammalian, the bat brain contains a sensory modality quite unlike any which we possess. By induction, we can guess that perhaps the difference between sonar perception could be as much as the difference between our visual and auditory perception. Yet, in some sense sonar is both visual and auditory, and still it is neither visual nor auditory. It is more similar to vision, because it helps build a model of the scene around us, however, instead of stereoscopic vision, the bat sonar can make accurate 3-D models of the environment from a particular point of view, in contrast with normal vision that is said to have "2-1/2D vision". Therefore, it is unlike anything that humans experience, and perhaps our wildest imaginations of bat sonar experience are doomed to fall short of the real thing. Namely because it is difficult for us to understand the experience of a detailed and perhaps rapidly updated 3-D scene that does not contain optical experience as there is no 2-D image data from eyes to be interpreted. This would likely require specialized neural circuitry. And despite what Nagel has in mind, it seems theoretically possible to "download" bat sonar circuitry into a human brain so that the human can experience the same sensory modality. This seems to be one of those things in which thinking alone is not sufficient. The only barrier to knowing what it is like to be bat is, thus, a technological barrier, not a conceptual or fundamental barrier.

    That being the case, we may also consider what an upload would experience, or whether it would experience anything, as brain uploading is a primary goal of transhumanism on which computational neuroscientists have already begun working. The question that I pose is harder because the upload usually does not run on a biological nervous system, and it is easier because the processing is the simulation of a human brain (and not something else). Answering this question is important, because presumably the (subjective) experience, the raw sensations and feelings of a functional human brain, are very personal and valuable to human beings. We would like to know, if there is a substantial loss or difference in the quality of experience for our minds' digital progeny.

    Brain prosthesis thought experiment

    The question is also very similar to the brain prosthesis thought experiment, in which biological neurons of a brain are gradually replaced by functionally  equivalent (same I/O behavior) synthetic (electronic) neurons. In that thought experiment, we ponder how the experience of the brain would change. As far as I can tell, Marvin Minsky and Hans Moravec think that nothing would change. And John R. Searle maintains that the experience would gradually vanish in his book The Rediscovery of the Mind. The reasoning of Minsky seems to be that it is sufficient for the entire neural computation to be  equivalent at the level of electrical signaling (as the synthetic neurons are electronic), while he seems to disregard other brain states. While for Searle, experience can only exist in "the right stuff", which he seems to be taking as biological substrate (although one cannot be certain) [4]. We will revisit this division of views soon enough.

    Naturalist theories of experience

    In a recent interview on H+, Ben Goertzel makes an intriguing summary of his views on "consciousness":

    Consciousness is the basic ground of the universe. It’s everywhere and everywhen (and beyond time and space, in fact). It manifests differently in different sorts of systems, so human consciousness is different from rock consciousness or dog consciousness, and AI consciousness will be yet different. A human-like AI will have consciousness somewhat similar to that of a human being, whereas a radically superhumanly intelligent AI will surely have a very different sort of conscious experience.

    While he does not explicitly state his views on this particular question, it seems that he would answer in a manner close to Minsky rather than Searle. Since the upload can be considered as a very human like AI, it seems that Goertzel anticipates that the experience of an upload will be somewhat similar to human. He also mentions that the basic stuff of consciousness must be everywhere, since our brains are formed from natural matter.

    Why is this point of view significant? The evidence from psychedelic drugs and anesthesia imply that changing the brain chemistry also modulates experience. If the experience changes, what can this be attributed to? Does the basic computation change, or are chemical interactions actually part of human experience? It seems that answering that sort of question is critical to answering the question posed in this article. However, it first starts with accepting that it is natural, like a star, or a waterfall. Only then can we begin to ask questions with more distinctive power.

    Over the years, I have seen that neuroscientists were almost too shy to ask these questions, as if these questions were dogma. Although no neuroscientist would admit to such a thing, of course, it makes me think whether religious or superstitious pre-suppositions may have a role in the apparent reluctance of neuroscientists to investigate this fundamental  question in a rigorous way. In one particular study, Bialek and his super-star team of cognitive scientists [2] may shed light on the question. There, Bialek's team makes the claim that the neural code forms the basis of experience, therefore changes in neural code (i.e. spike train, a spike train is the sequence of signals that travel down an axon), change experience. That's a very particular claim, that can be perhaps one day proven in experiment. However, at the present it seems like a hypothesis that we can work with, without necessarily accepting it.

    That is to say, we are going to analyze this matter in the framework of naturalism, without ever resorting to skyhooks. We can consider a hypothesis like Bialek's, however, we will try to distinguish finely between what we do know and what is hypothetical. Following this methodology, and a bit of common sense, I think we  can derive some scientifically plausible speculations, following the terminology of Carl Sagan.

    The debate

    Let's rewind a little. On one side, AI researchers (like Minsky) seem to think that uploading a mind will just work, and experience will be alright. On the other side, skeptics like Searle and Penrose, try everything to deny "consciousness" to poor machinekind.

    And on the other hand, Ray Kurzweil wittingly suggested that when the intelligent machines claim that they have conscious experience we will believe in them (because they are so smart and convincing). That goes without saying, of course, and human beings are gullible enough to believe in almost anything, but the question is rather, would a good engineer like himself be convinced? In all likelihood, I think that the priests and conservatives of this world will say that uploads have no "souls" and therefore they don't have the same rights as humans. And they will say that none of what the uploads say matters. Therefore, you have to have very good scientific evidence to show that this is not the case. If we leave this matter to superstitious people, they will find a way to twist it beyond our imagination.

    I'm hoping that I have convinced you that merely word play will not be sufficient. We need to have a good scientific theory of when and how experience occurs. The best theory will have to be induced from experimental neuroscience and related facts. What is the most basic criterion for assessing whether the theory of experience is scientifically sound? Well, no doubt, it comes down to rejecting each and any kind of supernatural/superstitious explanation and see this matter the same way as we are investigating problems in molecular biology, that the experience is ultimately made up of physical resources and interactions, and there is nothing else to it! In philosophy, this approach to mind is called "physicalism". A popular statement of physicalism is known as "token physicalism": "every mental state x is identical to a physical state y". That's something a neuroscientist can work with, because presumably, when the neuroscientist introduces a change to the brain, he would like to see a corresponding change in the mental state. One can think of cybernetic eye implants and transcranial magnetic stimulation and confirm that this holds in practice.

    Asking the question in the right way

    Now, we have every basic concept to frame the question in a way akin to analysis. Mental states are physical states. The brain states in a human constitute its subjective experience. The question is whether a particular whole brain simulation, will have experience, and if it does, how similar this experience is to the experience of a human being. If Ben Goertzel and I are right, then this is nothing special, it is a basic capability of every physical resource. However, we may question what physical states are part of human experience. We do not usually think that, for instance, the mitochondrial functions inside neurons, or the DNA, is part of the experience of the nervous system. We think like that, because they do not seem to be directly participating in the main function of the nervous system: thinking. Likewise, we don't really think that the power supply is part of the computation in a computer.

    This analogy might seem out of place, but it isn't. If Ben Goertzel and I are right, experience is one of the basic features of the universe. It's all around us, however, most of it is not organized in an intelligent way, and therefore we don't call them conscious. This is the simplest explanation of experience. It doesn't require any special stuff. Just "stuff" organized in the right way so as to yield an intelligent functional mind. Think of it like this. If today, some evil alien came and shuffled all the connections in your brain, would you still be intelligent? I think not. However, you should accept that even in that state, you would have an experience, an experience that is probably meaningless and chaotic, but an experience nonetheless. So, perhaps that's what a glob of plasma experiences.

    Neural code vs. neural states

    Let us now revisit the hypothesis of Bialek. Experience is determined by particular electrical signals. If that is true, even the experience of two humans is very different, because it has been shown that codes evolve in different ways. You can't just plug in the code from another human to someone else, it will be random to the second human. And if Bialek's right, it will be another kind of experience. Which basically means that the blue that I experience is different from the blue that you experience, while we presently have no way of directly comparing them. Weird as that may sound, as it is based on sound neuroscience research, it is a point of view we must take seriously.

    Yet even if the experiences of two humans can be very different, they must be sharing some basic quality or property of experience. Where does that come from? If experience is this complicated time evolution of electro-chemical signals, then it's the shared nature of these electro-chemical signals (and processing) that provides the shared computational platform. Remember that a change in the neural code (spike train) implies a lot of changes. For one thing, the chemical transmission across synapses would change. Therefore, even a brain prosthesis device that simulates all the electrical signaling insanely accurately, might still miss part of the experience, if the bio-chemical events that occur in the brain are part of experience.

    In my opinion, to answer the question decisively, we must first encourage the neuroscientists to attack the problem of human experience, and find the sufficient and necessary conditions for human experience to occur, or be transplanted from one person to the other. They should also find to what extent chemical reactions are important for experience. If, for instance, we find that the property of human experience crucially depends on quantum computations carried at synapses and inside neurons, that might mean that to construct the same kind of experience you would need similar material and method of computation.

    On the other hand, we need to consider the possibility that electrical signals may be a crucial part of experience, due to the power and information they represent, so perhaps any electronic device has these electron patterns that make up most of what you sense from the world around you. If that is true, the electronic devices presently would be assumed to contain human-like experience, for instance. Then, the precise geometry and connectivity of the electronic circuit could be significant. However, it seems to me that chemical states are just as important, and if as some people think quantum randomness plays some role in the brain, it may even be possible that the quantum description of the random number generator is relevant.

    Simulation and transforming experience

    At this point, you might be wondering if the subject was not simulation. Is the question like whether the simulation of rain is wet? In some respects, it is, because obviously, the simulation of wetness on a digital computer is not wet in the ordinary sense. Yet, a quantum-level simulation that affords all the subtleties of chemical and molecular interactions can be considered such. I suppose that, we can invoke the concept of a "universal quantum computer" from theory, and claim that a universal quantum computer would indeed re-instate wetness, in some sort of a "miniature pocket universe". Even that is of course very much subject to debate (as you can follow from the little digression on philosophy I provide at the end of the article).

    With all the confusing things that I have said, it might appear now that we know less than we started out with. However, this is not the case. We have a human brain A, a joyous lump of meat, and its digitized form B, running on a digital computer. Will B's experience be the same as A's, or different, or non-existent?

    Up to now, if we accept the simplest theory of experience (that it requires no special conditions to exist at all!), then we conclude that B will have some experience, but since the physical material is different, it will have a different texture to it. Otherwise, an accurate simulation, by definition, holds the same organization of cognitive constructs, like perception, memory, prediction, reflexes, emotions, etc., accurately, and since the dreaded panpsychism is accepted to be correct, they will give rise to an experience "somewhat similar to the human brain" as Ben Goertzel said about human-like AI's, yet the computer program B, may be experiencing something else at the very lowest level. Simply because it's running on some future nanoprocessor instead of the brain, the physical states have become altogether different, yet their relative relationship, i.e. the structure of experience, is preserved.

    Let us try to present the idea here more intuitively. As you know, the brain is some kind of an analog/biological computer. A great analogy is the transfer of a 35mm film to a digital-format. Surely, many critics have held that the digital format will be ultimately inferior, and indeed the texture is different but the (film-free) digital medium also has its affordances like being able to backup and copy easily. Or maybe we can contrast an analog sound synthesizer with a digital sound synthesizer. It's difficult to simulate an analog synthesizer, but you can do it to some extent. However, the physical make-up of an analog synthesizer and digital synthesizer are quite different. Likewise, B's experience will have a different physical texture but its organization can be similar, even if the code of the simulation program of B will necessarily introduce some physical difference (for instance neural signals can be represented by a binary code rather than a temporal analog signal). So who knows, maybe the atoms and the fabric of B's experience will be different altogether as they are made up of the physical instances of computer code running on a universal computer, as improbable as it may seem, these people are made up of live computer codes, so it would be naive to expect that their nature will be the same as ours. In all likelihood, our experience would necessarily involve a degree of unimaginable features for them, as they are forced to simulate our physical make-up in their own computational architecture. This brings a degree of relative dissimilarity as you can see. And other physical differences only amplify this difference.

    Assuming the above explanation, therefore, when they are viewing the same scene, both A and B will claim to be experiencing the scene as they always did, and they will additionally claim that no change has occurred since the non-destructive uploading operation went successfully. This will be the case, because the state of experience is more akin to the RAM of computers. It's this complex electro-chemical state that is held in memory with some effort, by making the same synapses repeat firing consistently, so that more or less the same physical state is maintained. This is what must be happening when you remember something, a neural state that is somewhat similar to when the event happened should be created. Since in B, the texture has changed, the memory will be re-enacted in a different texture, and therefore B will have no memory of what it used to feel like being A.

    Within the general framework of physicalism, we can comfortably claim that further significant changes will also influence B's experience. For instance, it may be a different thing to work on hardware with less communication latency. Or perhaps if the simulation is running on a very different kind of architecture, then the physical relations may change (such as time and geometry) and this may influence B's state further. We can imagine this to be asking what happens when we simulate a complex 3-D computer architecture on a 2-D chip.

    Moreover, a precise answer seems to depend on a number of smaller questions that we have little knowledge or certainty of. These questions can be summarized as:

    1. What is the right level of simulation for B to be functionally equivalent to A? If certain bio-chemical interactions are essential for the functions of emotions and sensations (like pleasure), for instance, then not simulating them adequately would result in a definite loss of functional accuracy. B would not work the same way as A. This is true even if spike trains and changes in neural organization (plasticity) are simulated accurately. It is also unknown whether we can simulate at a higher level, for instance via Artificial Neural Networks, that have abstracted the physiological characteristics altogether and just use numbers and arrows to represent A. It is important to know these so that B does not turn out to be an emotionless psychopath.
    2. How much does the biological medium contribute to experience? This is one question that most people avoid answering because it is very difficult to characterize. The most general characterizations may use algorithmic information theory or quantum information theory. However, in general, we may say that we need an appropriate physical and informational framework to answer this question in a satisfactory manner. In the most general setting, we can claim that ultimately low-level physical states must be part of experience, because there is no alternative.
    3. Does experience crucially depend on any funky physics like quantum coherence? Some opponents of AI, most notably Penrose [5], have held that "consciousness" is due to macro-level quantum phenomena, by which they try to explain "unity of experience". While on the other hand, many philosophers of AI think that the unity is an illusion. Yet, the illusion is something to explain, and it may well be that certain quantum interactions may be necessary for experience to occur, much like superconductivity. This again seems to be a scientific hypothesis, which can be tested.

    I think that the right attitude to answering these finer questions is again a strict adherence to naturalism. For instance, in 3, it may seem easier to also assume a semi-spiritualist interpretation of Quantum Mechanics, and claim that the mind is a mystical soul. That kind of reasoning will merely help to stray away from scientific knowledge.

    I am hoping that you see the panpsychism approach is actually the simplest theory of experience, that everything has experience. Then, when we ask a physicist to quantify that, she may want to measure the energy, or the amount of computation or communication, or information content, or heat. Something that can be defined precisely, and worked with.  I suggest that we use such methods to clarify these finer questions. Thus, assuming the generalist theory of panpsychism, I can attempt to answer the above finer questions. At this point, since we do not have conclusive scientific evidence, this is merely guesswork, and I'm  going to give conservative answers. My answer to 1. could for instance be at the level of molecular interactions which would at least cover the differences among various neurotransmitters, and which we can simulate on digital computers (perhaps imprecisely, though). The answer to 2. is at least as much as required for correct functionality, and at most all the information as present in the biological biochemistry (i.e. precise cellular simulations). This might be significant in addition to electrical signals. And to 3. Not necessarily. According to panpsychism, it may be claimed to be false, since it would constrain minds to funky physics (and contradict with the main hypothesis). If, for instance, quantum coherence is indeed prevalent in the brain and provides much of the "virtual reality" of the brain, then the panpsychist could argue that quantum coherence is everywhere around us. Indeed, we may have a rather primitive understanding of coherence/decoherence yet, as that is itself one of the unsettled controversies in philosophy of physics. For instance, one may question what happens if the wave function collapse is deterministic as in Many Worlds Interpretation.

    Other finer points of inquiry may as well be imagined, and I would be delighted to hear some samples from the readers. These finer questions illustrate the distinctions between specific positions, therefore the answers could also be quite varied, no doubt.

    After these closing remarks, comes a section reminiscing the fiery philosophical background of this article.

    Infinite philosophical regression

    The philosophy behind this article goes a long way of arguing over and over again about basic statements of cosmology, physics, computation, information and psychology. It is not certain how fruitful that approach has become. Yet for the sake of completeness, I wish to give some further references to follow. For philosophy of mind in general, Jaegwon Kim's excellent textbook on the subject will provide you with enough verbal ammunition to argue endlessly for several years to come. That is not to say that philosophical abstraction cannot be useful. It can guide the very way we conduct science. However, if we would like that useful outcome, we must pay a lot of attention to fallacies that have plagued philosophy with many superstitious notions. For instance, we should not let religion or folk psychology much into our thoughts. Conducting thought experiments is very important, but they should be taken with care so that the thought experiment would actually be possible in the real world, even though it is very difficult or practically impossible to realize. For that reason, per ordinary philosophical theories of "mind", I go no further than neuro-physiological identity theory, which is a way of saying that your mind is literally the events that happen in your brain. Rather than being something else like a soul, a spirit, or a ghost. The reader may have also noticed that I have not used the word "qualia" because of its somewhat convoluted connotations. I did talk about the quality of experience, which is something you can think about. In all the properties that can be distinguished in this fine experience of having a mind, maybe some of them are luxurious even; and that's why I used the word "quality" rather than "qualia" or "quale".

    About the sufficient and necessary physical conditions, I've naturally spent some time exploring the possibilities. I think it is quite likely that quantum interactions may be required for human experience to have the same quality as an upload's, since biology seems inventive in making use of quantum properties, more than we thought, and as I suppose you would remember because macro bio-molecules have been shown to have quantum behavior. Maybe, Penrose is right. That is possible. However, specific experiments would have to be conducted to demonstrate it.  I can see why computational states would evolve, but not necessarily why they would have to depend on macro-scale quantum states, and I don't see what this says precisely on systems that do not have any quantum coherence. Beyond Penrose, I think that the particular texture of our experience may indeed depend on chemical states, whether  quantum coherence is involved or not. If of course the brain turned out to be a quantum-computer under our very noses, that would be fantastic and we could then emulate the brain states very well on artificial quantum computers. In this case, assuming that the universal quantum computer itself has little overhead, the quantum states of the upload could very well closely resemble the original.

    Other physical conditions can be imagined as well. For instance, digital physics provides a comfortable framework to discuss experience. The psychological patterns would be cell patterns in the  universal cellular automata.  A particular pattern may describe a particular experience. Then, two patterns are similar to the extent they are syntactically similar. Which would mean that, you still cannot say that the upload's experience will be the same. It will likely be quite different.

    One of my nascent theories is the Relativistic Theory of Mind, it is discussed in an ai-philosophy mailing list thread, which obviously tries to explain subjectivity of experience with concepts from the theory of relativity. From that point of view, it makes sense that different energy distributions have different experience, since measurements change.

    I think that a general description of the difference between two systems can be captured by algorithmic information theory (among others perhaps). I have previously applied it to the reductionism vs. non-reductionism debate in philosophy [3]. I think that debate stems mainly from disregarding the mathematics of complexity and randomness. As part of ongoing research, I am making some effort to apply it to problems in philosophy. Here, it might correspond to saying that the similarity between A's and B's states depends on the amount of mutual information in the physical make-up of A, and  the physical make-up of B. As a consequence, the dissimilarity between two systems would be only the informational difference in the low-level physical structures of A and B,  together with the information of the simulation program (not present in A at all), which could be quite a bit if you compare nervous systems and electronic computer chips running a simulation. Perhaps, this difference is not so insignificant that it will not have an important contribution to experience.

    Please also note that the view presented here is entirely different from Searle, who seemed to have a rather vitalist attitude towards the problem of mind. According to him, the experience vanishes, because it's not the right stuff, which seems to be the specific biochemistry of the brain for him [4]. Regardless of the possibility of an artificial entity to have the same biochemistry, this is still quite restrictive. Some people call it carbon-chauivinism, but I actually think it's merely idolization of earth biology, as if it is above everything else in the universe.

    And lastly, you can participate in the discussion of this issue on the corresponding ai-philosophy thread.


    1. Thomas Nagel, 1974, "What Is it Like to Be a Bat?", Philosophical Review, pp. 435-50.

    2. E Schneidman, N Brenner,N Tishby, RR de Ruyter van Steveninck, & W Bialek, 2001, "Universality and individuality in a neural code"., In Advances in Neural Information Processing 13, TK Leen, TG Dietterich & V Tresp, eds, pp 159–165 (MIT Press, Cambridge, 2001); arXiv:physics/0005043 (2000).

    3. Eray Özkural, 2005, "A compromise between reductionism and non-reductionism", In WORLDVIEWS, SCIENCE AND US Philosophy and Complexity,University of Liverpool, UK, 11 – 14 September 2005. World Scientific Books, 2007.

    4. John Searle, 1980, “Minds, Brains, and Programs.” Behavioral and Brain Sciences 3, 417-424.

    5. Hameroff, S.R. and Penrose, R., 1996,  "Orchestrated reduction of quantum coherence in brain microtubules: a model for consciousness ? " In Toward a Science of Consciousness - The First Tucson Discussions and Debates, eds. Hameroff, S.R., Kaszniak, A.W., and Scott, A.C., Cambridge, MA: MIT Press, pp.507-540.

    Jan 312011

    Walker.'s original article.

    Oh well, Mark, I think you want us to feel really joyful! Wake up with a surge of happiness in the morning, have a great breakfast, go to work, work, lunch, work, work, until we drop dead, happy as Jim Morrison when he did!

    No, I'm not going to write a distressed blog entry. After all, stress is a disease. Constantly worrying about things that do not really interest us, that must be a disease. Why should we be constantly worried about if we are being screwed by politicians, our boss, our family? That creates unrest and disorder. It can just bring more unhappiness to this bleak, cold world. What could be better than making non-unhappy individuals giddy and laughing all the time, until their soft bellies crack and their internal organs spread on the floor?

    Well I don't think so.  With these happy pills, every sheep will be happy to be oppressed by fascist jocks.

    You are already being given lots of "happy pills". TV/news/movies/music, all this infoshit are indeed superior to many other chemical kinds of happy pills. It is how they can drag people across continents to kill for oil, and make themselves feel proud for it. A pharmacologically potent happy pill would be a disaster for transhumanism. The governments would be so happy to just let people have a better version of Prozac!!!!! Double-plus ultra happy, you morons!