Imagine standing at the college dining hall, faced with tens of options but holding only a small(ish) plate. How do you choose? In this course, we discuss how trial-and-error learning gives rise to the everyday behavior of humans and other animals. We take a modern, integrative approach to phenomena that grew from animal-learning paradigms such as classical conditioning and instrumental conditioning, and examine them through the lens of computational models of learning and decision making and current neuroscientific knowledge. We answer questions such as: why do we have such a hard time walking away from rewards (and how is this exploited by fortune tellers?), how best can we train a dog to do new tricks (and how does this apply to raising children?), why do we sometimes find ourselves performing an action out of habit, even though we have no desire for its outcome (e.g., opening the fridge as we walk into the kitchen, even though we are not hungry), why do we press the “walk” button more times when we are in a hurry (surely once is enough, no?), do we learn differently when we are a bad mood (and do thing we learn affect our mood?),  and how do we optimally teach someone something new so that we can facilitate them changing their mind.

For each topic, we discuss behavioral and psychological findings, computational algorithms that underlie this behavior, and the neural hardware (and software) that may be implementing the algorithms in the brain. The overarching goal is to develop an integrative picture of how the brain implements the computations that are necessary in order to bring about day-to-day behavior as we know it. In the second half of the course, students work with peers on a project investigating a topic of their choice.

This course follows an alternative grading/teaching without grades approach to learning (see more information below).

Learning objectives: Upon completion of this course, students should be able to

Recognize the multiple mechanisms that can bring about a decision to behave, and understand their essential differences.
Describe in computational terms how new information leads to learning in each system.
Work in a group to independently research a project to pursue your own curiosity about learning and decision making.

Course topics and presentations (slides to be added throughout the semester)*
*Many thanks to all my colleagues and fellow teachers who have (often unknowingly) contributed much of the material that is included in the above slides. If you are teaching a course and would like to use these materials in your own slides, feel free to email me for the keynote or powerpoint version at yael@princeton.edu

Prologue: Introduction (slides)

Act 1: Classical/Pavlovian conditioning (weeks 1-3) – basic phenomena (slides), learning from prediction errors (slides) and the Resorla-Wagner model (flipped portion + slides), second order conditioning and the Temporal Difference model (flipped portion + slides), dopamine and learning in the basal ganglia (slides), opponent processes: appetitive vs. aversive, inhibitory vs. excitatory conditioning (slides)

Act 2: Instrumental/operant conditioning (weeks 4-6) – Thorndike, basic procedures, modeling action selection, Actor/Critic framework, neural substrates of Actor/Critic model (slides), learning Q values, comparing models using neural data (slides), Skinner & free operant behavior , modeling free operant behavior (slides), habits (model-free learning) versus goal-directed behavior (model-based learning) (slides), everything I taught you so far is somewhat wrong (slides), so…

Act 3: Choose your own adventure
Bayesian inference+POMDPs (slides), latent cause inference, extinction, and learning state representations (slides), Changing people’s minds and behavior (recording), Computational psychiatry: mood and reinforcement learning (slides)

Epilogue: What have we learned? (slides)

Student projects from previous years

More about “teaching without grades”
Learning is not a spectator sport – it is a choice, and requires commitment. There is also no single metric for learning that is relevant to everyone. Like a muscle, our brain can always expand in its abilities, through practice and training. To maximize learning, research shows that goals that are just beyond a person’s current abilities are most motivating, and promote growth. As we see in this course, research also shows that grades (for humans) and rewards (for animals) can be detrimental to learning. In particular, focusing on externally-assigned (and somewhat arbitrary) goals, and worrying about whether you are achieving them, does little to promote true, deep, and consistent learning.

So we do not do that. Instead, in this course students set personal goals, track their own progress on these goals throughout the semester, amend their goals as needed (setting new goals when previous ones have been achieved; modifying goals that are not realistic), and assess their your own learning. The course instructors help students achieve their goals for each course component by requesting students to reflect on their progress often, and discussing with them challenges that arise as they work toward their individual goals, how to measure and monitor their progress, etc. We also provide feedback on all course components, to help guide students in improving in any aspect in which they are attempting growth, but without narrowing that feedback to a one-dimensional number or letter. Students’ abilities, challenges, and learning are multidimensional. We have found that providing feedback on these many dimensions is much more useful for promoting growth than are summary grades.

For more information about alternative grading I recommend this excellent episode of Jennifer Gonzalez’s fantastic Cult of Pedagogy, this practical how-to book by Starr Sackstein, and here is a twitter thread where I wrote about this method and discussed it with the community.

Assigned reading (from 2022):

Rescorla (1988) – Pavlovian conditioning: It’s not what you think it is
Tobler, Dickinson & Schultz (2003) – Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm
Bonawitz et al (2011) – The double-edged sword of pedagogy: Instruction limits spontaneous exploration and discovery

Supplementary/optional reading:

Niv (2018) – Deep down, you are a scientist
Sutton & Barto (1990) – Time derivative models of Pavlovian reinforcement
Barto (1995) – Adaptive critics in the basal ganglia
Niv & Schoenbaum (2008) – Dialogues on prediction errors
Niv (2009) – Reinforcement learning in the brain
Tolman (1948) – Cognitive maps in rats and men
Drummond & Niv (2020) – Model-based decision making and model-free learning (a primer)
Dijksterhuis et al (2007) – On making the right choice: The deliberation without attention effect

Additional resources

Gluck, Mercado & Myers – Learning and Memory: From Brain to Behavior – As close to our textbook as there is, though it covers more than we discuss (eg, memory) and does not cover everything that we discuss (eg, computational models)
Sutton, R. and Barto, A. – Reinforcement Learning (2018) – A very good and accessible book explaining the computational field of reinforcement learning (also available online)
Mackintosh, N.J. – The Psychology of Animal Learning (1974) – Great book, although hard to find (out of print).
Mackintosh, N.J. – Conditioning and Associative Learning (1983) – Shorter than the above and covers mostly theoretical aspects, not for complete beginners.
Dickinson, A. – Contemporary Animal Learning Theory (1980) – Although this was contemporary a long time ago, it is still very good and easy to read.
Classics in the History of Psychology – Classic papers in psychology ONLINE (as well as information about their authors)
Behavioral Science Glossary – A useful glossary of many of the terms we use
PsychWeb – Resource for psychology links
Sniffy the Virtual Rat – 20 day downloadable demo of Sniffy