+ Site Statistics
+ Search Articles
+ PDF Full Text Service
How our service works
Request PDF Full Text
+ Follow Us
Follow on Facebook
Follow on Twitter
Follow on LinkedIn
+ Subscribe to Site Feeds
Most Shared
PDF Full Text
+ Translate
+ Recently Requested

The emergence of saliency and novelty responses from Reinforcement Learning principles



The emergence of saliency and novelty responses from Reinforcement Learning principles



Neural Networks 21(10): 1493-1499



Recent attempts to map reward-based learning models, like Reinforcement Learning [Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An introduction. Cambridge, MA: MIT Press], to the brain are based on the observation that phasic increases and decreases in the spiking of dopamine-releasing neurons signal differences between predicted and received reward [Gillies, A., & Arbuthnott, G. (2000). Computational models of the basal ganglia. Movement Disorders, 15(5), 762-770; Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80(1), 1-27]. However, this reward-prediction error is only one of several signals communicated by that phasic activity; another involves an increase in dopaminergic spiking, reflecting the appearance of salient but unpredicted non-reward stimuli [Doya, K. (2002). Metalearning and neuromodulation. Neural Networks, 15(4-6), 495-506; Horvitz, J. C. (2000). Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience, 96(4), 651-656; Redgrave, P., & Gurney, K. (2006). The short-latency dopamine signal: A role in discovering novel actions? Nature Reviews Neuroscience, 7(12), 967-975], especially when an organism subsequently orients towards the stimulus [Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80(1), 1-27]. To explain these findings, Kakade and Dayan [Kakade, S., & Dayan, P. (2002). Dopamine: Generalization and bonuses. Neural Networks, 15(4-6), 549-559.] and others have posited that novel, unexpected stimuli are intrinsically rewarding. The simulation reported in this article demonstrates that this assumption is not necessary because the effect it is intended to capture emerges from the reward-prediction learning mechanisms of Reinforcement Learning. Thus, Reinforcement Learning principles can be used to understand not just reward-related activity of the dopaminergic neurons of the basal ganglia, but also some of their apparently non-reward-related activity.

Please choose payment method:






(PDF emailed within 0-6 h: $19.90)

Accession: 056329194

Download citation: RISBibTeXText

PMID: 18938058

DOI: 10.1016/j.neunet.2008.09.004


Related references

Effects of stimulus novelty and dimensional saliency in human shift learning. Journal of Experimental Psychology 98(2): 264-270, 1973

Novelty is not enough: laser-evoked potentials are determined by stimulus saliency, not absolute novelty. Journal of Neurophysiology 109(3): 692-701, 2013

The effect of novelty on reinforcement learning. Progress in Brain Research 202: 415-439, 2013

Novelty and Inductive Generalization in Human Reinforcement Learning. Topics in Cognitive Science 7(3): 391-415, 2015

Altered responses to novelty and drug reinforcement in adult rats treated neonatally with domoic acid. Physiology and Behavior 93(1-2): 327-336, 2008

Emergence of linguistic conventions in multi-agent reinforcement learning. Plos one 13(11): E0208095, 2018

Using reinforcement learning to understand the emergence of "intelligent" eye-movement behavior during reading. Psychological Review 113(2): 390-408, 2006

Association between locomotor response to novelty and light reinforcement: sensory reinforcement as a rodent model of sensation seeking. Behavioural Brain Research 230(2): 380-388, 2012

A model to explain the emergence of reward expectancy neurons using reinforcement learning and neural network. Neurocomputing 69(10-12): 1327-1331, 2006

Imaginative Reinforcement Learning: Computational Principles and Neural Mechanisms. Journal of Cognitive Neuroscience 29(12): 2103-2113, 2017

Novelty and saliency in attentional capture by unannounced motion singletons. Acta Psychologica 136(3): 290-299, 2011

Contrasting temporal difference and opportunity cost reinforcement learning in an empirical money-emergence paradigm. Proceedings of the National Academy of Sciences of the United States of America 115(49): E11446-E11454, 2018

Dissociating intentional learning from relative novelty responses in the medial temporal lobe. Neuroimage 25(1): 51-62, 2005

From Creatures of Habit to Goal-Directed Learners: Tracking the Developmental Emergence of Model-Based Reinforcement Learning. Psychological Science 27(6): 848-858, 2016

How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. European Journal of Neuroscience 35(7): 1024-1035, 2012