A fascinating hypothesis is that human and animal intelligence could be explained by a few principles (rather than an encyclopedic list of heuristics). If that hypothesis was correct, we could more easily both understand our own intelligence and build intelligent machines.

Just like in physics, the principles themselves would not be sufficient to predict the behavior of complex systems like brains, and substantial computation might be needed to simulate human-like intelligence.

This hypothesis would suggest that studying the kind of inductive biases that humans and animals exploit could help both clarify these principles and provide inspiration for AI research and neuroscience theories.

Deep learning already exploits several key inductive biases, and this work considers a larger list, focusing on those which concern mostly higher-level and sequential conscious processing.

The objective of clarifying these particular principles is that they could potentially help us build AI systems benefiting from humans' abilities in terms of flexible out-of-distribution and systematic generalization, which is currently an area where a large gap exists between state-of-the-art machine learning and human intelligence.

Is 100% accuracy on the test set enough? Many machine learning systems have achieved excellent accuracy across a variety of tasks, yet the question of whether their reasoning or judgment is correct has come under question, and answers seem to be wildly inconsistent, depending on the task, architecture, training data, and interestingly, the extent to which test conditions match the training distribution.

Have the main principles required for deep learning to achieve human-level performance been discovered, with the main remaining obstacle being to scale up? Or do we need to follow a completely different research direction not built on the principles discovered with deep learning, in order to achieve the kind of cognitive competence displayed by humans?

Our goal here is to better understand the gap between current deep learning and human cognitive abilities so as to help answer these questions and suggest research directions for deep learning with the aim of bridging the gap towards human-level AI.

Our main hypothesis is that deep learning succeeded in part because of a set of inductive biases (preferences, priors or assumptions), but that additional ones should be included in order to go from good in-distribution generalization in highly supervised learning tasks (or where strong and dense rewards are available), such as object recognition in images, to strong out-of-distribution generalization and transfer learning to new tasks with low sample complexity (few examples needed to generalize well).

To make that concrete, we consider some of the inductive biases humans may exploit in conscious thought using highly sequential cognition operating at the level of conscious processing, and review some early work exploring these "high-level cognitive inductive priors" in deep learning.

We use the term high-level to talk about variables that are manipulated at the conscious level of processing and are thus generally verbalizable. However, humans can consciously focus attention on low-level or intermediate-level features, e.g., by describing an odd-coloured pixel, not just very abstract concepts like objects or social situations.

We argue that the deep learning progression from MLPs to convnets to transformers has in many ways been an (incomplete) progression towards the original goals of deep learning, i.e., to enable the discovery of a hierarchy of representations, with the most abstract ones, often associated with language, at the top.

Note however, that although language may give us a view on system 2, these abilities are likely to pre-exist language as there is evidence of surprisingly strong forms of on-the-fly reasoning in some non-human animals, like corvids.

Our arguments suggest that while deep learning brought remarkable progress, it needs to be extended in qualitative and not just quantitative ways: larger and more diverse datasets and more computing resources are important but insufficient without additional inductive biases.

We make the case that evolutionary forces, the interactions between multiple agents, the non-stationary and competition systems put pressure on the learning machinery to achieve the kind of flexibility, robustness and ability to adapt quickly which humans seem to have when they are faced with new environments but needs to be improved with deep learning.

The sought-after inductive biases should thus especially help AI to progress on these fronts. In addition to thinking about the learning and sample complexity advantage of these inductive biases, this paper links them with knowledge representation in neural networks, with the idea that by decomposing knowledge into its stable parts (like causal mechanisms) and volatile parts (random variables), and factorizing knowledge in small and somewhat independent pieces that can be recomposed dynamically as needed (to reason, imagine or explain at an explicit and verbalizable level), one may achieve the kind of systematic generalization which humans enjoy and is common in natural language.