Human Compatible: Artificial Intelligence and the Problem of Control

Intelligence, as we understand it, is not a mystical gift but a deeply practical capability: the ability to act in ways that are likely to achieve our objectives, given what we perceive. This principle is visible even in the simplest forms of life. Consider the humble bacterium, E. coli, which navigates its watery environment using tiny rotating flagella. It swims in a seemingly random pattern, alternating between swimming straight and tumbling. Yet, this randomness is tempered: when it senses increasing concentrations of glucose, its preferred energy source, it swims longer and tumbles less, effectively steering itself toward nourishment. This behavior exemplifies intelligence as goal-directed action based on perception.

Such simple organisms do not possess brains or memories, but evolution has endowed them with mechanisms that allow their behavior to vary according to environmental cues, providing a basic form of decision-making. This insight extends to us humans, whose complex brains are built upon similar principles, but with vastly greater capacity. Our brains contain billions of neurons connected by synapses, whose strengths change with experience, enabling learning. Central to this learning is the reward system, mediated by dopamine, which signals positive and negative outcomes, guiding behavior to maximize long-term benefit. This biological reward system parallels the reinforcement learning techniques used in modern AI.

Moreover, learning itself accelerates evolution through what is known as the Baldwin effect. Instead of relying solely on genetic changes, organisms that can learn adapt more quickly within their lifetimes, easing evolutionary pressures. This effect helps explain the emergence of complex brains and sophisticated intelligence across species.

In summary, intelligence arises from the interplay of perception, decision-making, and learning, all aimed at fulfilling objectives. This understanding grounds our approach to artificial intelligence: machines should act to achieve human objectives based on uncertain and incomplete perceptions, just as living beings do. As we embark on this journey into AI, we carry with us the wisdom encoded in nature's earliest learners.

With this foundation in place, we can now explore how rationality formalizes intelligence and decision-making, setting the stage for machines that reason and act under uncertainty.