While most approaches to creating AGI are theoretically motivated in one way or another, nobody would claim there currently exists a thorough and systematic theory of AGI in the same sense that there exist theories of say, sorting algorithms, respiration, genetics, or near-equilibrium thermodynamics. Current AGI theory is a patchwork of overlapping concepts, frameworks and hypotheses, often synergetic and sometimes mutually contradictory. Current AGI system designs are usually inspired by theories, but do not have all their particulars derived from theories. The creation of an adequate theoretical foundation for AGI is far beyond the scope of this review paper; however, it does seem worthwhile to briefly comment on what we may hope to get out of such a theory once it has been developed. Or in other words: What might a general theory of general intelligence look like?

Some of the things AGI researchers would like to do with a general theory of general intelligence are: • Given a description of a set of goals and environments (and perhaps a probability distribution over these), and a set of computational resource restrictions, determine what is the system architecture that will display the maximum general intelligence relative to these goals and environments, subject to the given restrictions • Given a description of a system architecture, figure out what are the goals and environments, with respect to which it will reach a relatively high level of general intelligence • Given an intelligent system architecture, determine what sort of subjective experience the system will likely report having, in various contexts • Given a set of subjective experiences and associated environments, determine what sort of intelligent system will likely have those experiences in those environments • Find a practical way to synthesize a general-intelligence test appropriate for a given class of reasonably similar intelligent systems • Identify the implicit representations of abstract concepts, arising within emergentist, hybrid, program learning based or other non-wholly-symbolic intelligent systems • Given a certain intelligent system in a certain environment, predict the likely course of development of that system as it learns, experiences and grows • Given a set of behavioral constraints (for instance, ethical constraints), estimate the odds that a given system will obey the constraints given certain assumptions about its environment. Determine architectures that, consistent with given computational resource constraints, provide an optimal balance between general intelligence for specified goals and environments, and adherence to given behavioral constraints • What are the key structures and dynamics required for an AGI system to achieve human-level, human-like general intelligence within feasible computational resources? • Predict the consequences of releasing an AGI into the world, depending on its level of intelligence and some specificities of its design • Determine methods of assessing the ethical character of an AGI system, both in its current form and in future incarnations likely to develop from its current form

Anyone familiar with the current state of AGI research will find it hard to suppress a smile at this ambitious list of objectives. At the moment we would seem very far from having a theoretical understanding capable of thoroughly addressing any of these points, in a practically useful way. It is unclear to how far the limits of mathematics and computing will allow us to progress toward theoretical goals such as these. However: the further we can get in this direction, the better off the AGI field will be. At the moment, AGI system design is as much artistic as scientific, relying heavily on the designer's scientific intuition. AGI implementation and testing are interwoven with (more or less) inspired tinkering, according to which systems are progressively improved internally as their behaviors are observed in various situations. This sort of approach is not unworkable, and many great inventions have been created via similar processes. It's unclear how necessary or useful a more advanced AGI theory will be for the creation of practical AGI systems. But it seems likely that, the further we can get toward a theory providing tools to address questions like those listed above, the more systematic and scientific the AGI design process will become, and the more capable the resulting systems.

It's possible that a thorough, rigorous theory of AGI will emerge from the mind of some genius AGI researcher, in one fell swoop – or from the mind of one of the early AGI successes itself! However, it appears more probable that the emergence of such a theory will be a gradual process, in which theoretical and experimental developments progress hand in hand.