A critic in action? A functional examination of the Striato-Pallido-Habenular circuit

Abstract

The basal ganglia (BG) and the midbrain dopamine (DA) system are considered key loci of reinforcement learning (RL), or learning by "trial and error", in the brain. The BG are implicated in action selection, and thus the "trial" part of the learning process, and the dopamine (DA) system is known to encode "error" signals. This DA error signal—the reward prediction error—is thought to adjust the BG’s propensity to select a "tried" action again in the future. In RL terms, the action-selecting BG is called the "actor", and the action-critiquing DA system the "critic". Here, a candidate striato-pallido-habenular "critic pathway" upstream of the DA system is examined.

The proposed critic pathway originates in the striosome compartment of the striatum, and projects via a non-canonical internal globus pallidus (GPi) population to the lateral habenula (LHb). LHb activity has been shown to encode the inverse of the DA reward prediction error signal, and to cause inhibition within the DA system. This posits the described striato-pallidohabenular pathway as key part of the critic circuit.

To investigate the role of the striato-pallido-habenular pathway in action, we recorded and manipulated the neuronal activity of neurons of the striatal striosome (Article I) and the GPi (Article II & III) in mice performing tasks that engendered trial and error behavioral strategies. We found that the activity of striatal striosome neurons had much in common with that of neurons within the striatal "actor pathways". Moreover, all striatal neurons jointly represented the evolving behavioral context in a spatiotemporally continuous population code, undermining notions of discrete and well-de ned action selection and evaluation processes. The results of our experiments on the GPi challenged its proposed role in driving the LHb’s inverse reward prediction error signals, and implicated the GPi-adjacent lateral hypothalamus (LHA) in that role instead. In sum, the studies included here call into question whether the striato-pallido-habenular pathway serves as a critic in BG-mediated action.