Imagine standing at the college dining hall, faced with tens of options but holding only a small(ish) plate. How do you choose? In this course, we discuss how trial-and-error learning gives rise to the everyday behavior of humans and other animals. We take a modern, integrative approach to phenomena that grew from animal-learning paradigms such as classical conditioning and instrumental conditioning, examining them through the lens of computational models of learning and decision making and current neuroscientific knowledge. We answer questions such as: why do we have such a hard time walking away from rewards (and how is this exploited by fortune tellers?), how best can we train a dog to do new tricks (and how does this apply to bringing up children?), why do we sometimes find ourselves performing an action out of habit, even though we have no desire for its outcome (e.g., opening the fridge as we walk into the kitchen, even though we are not hungry), why do we press the “walk” button more times when we are in a hurry (surely once is enough, no?), and how do we optimally teach someone something new so that we can facilitate them changing their mind.

For each topic, we discuss behavioral and psychological findings, computational algorithms that underlie this behavior, and the neural hardware (and software) that may be implementing the algorithms in the brain. The overarching goal is to develop an integrative picture of how the brain implements the computations that are necessary in order to bring about day-to-day behavior as we know it. In the second half of the course, you will work with peers on a project investigating a topic of your choice.

This course follows an alternative grading/teaching without grades approach to learning (see more information below).

Learning objectives: Upon completion of this course, students should be able to

  1. Recognize the multiple mechanisms that can bring about a decision to behave, and understand their essential differences.
  2. Describe in computational terms how new information leads to learning in each system.
  3. Work in a group to independently research a project to pursue your own curiosity about learning and decision making.

Course topics and presentations*

*Many thanks to all my colleagues and fellow teachers who have (often unknowingly) contributed much of the material that is included in the above slides. If you are teaching a course and would like to use these materials in your own slides, feel free to email me for the keynote or powerpoint version at yael@princeton.edu
  • Prologue: Introduction (slides)
  • Act 1: Classical/Pavlovian conditioning (weeks 1-3)basic phenomena (slides), learning from prediction errors (slides) and the Resorla-Wagner model (flipped portion + slides), second order conditioning and the Temporal Difference model (flipped portion + slides), dopamine and learning in the basal ganglia (slides), opponent processes: appetitive vs. aversive, inhibitory vs. excitatory conditioning (slides)
  • Act 2: Instrumental/operant conditioning (weeks 4-6) Thorndike, basic procedures, modeling action selection, Actor/Critic framework, neural substrates of Actor/Critic model (slides), learning Q values, comparing models using neural data (slides), Skinner & free operant behavior , modeling free operant behavior (slides), habits (model-free learning) versus goal-directed behavior (model-based learning) (slides), everything I taught you so far is somewhat wrong (slides), so…
  • Act 3: Choose your own adventure 
    • (week 7) Bayesian inference, POMDPs (slides), latent cause inference and learning state representations (slides)
    • (weeks 8-10) Student project presentations
    • (week 11) Changing people’s minds and behavior
    • (week 12) Computational psychiatry: mood and reinforcement learning
  • Epilogue: What have we learned? 

Student projects from previous years

 

More about “teaching without grades”

Learning is not a spectator sport – it is a choice, and requires commitment. There is also no single metric for learning that is relevant to everyone. Like a muscle, our brain can always expand in its abilities, through practice and training. To maximize learning, research shows that goals that are just beyond your current abilities are most motivating, and promote growth. As we see in this course, research also shows that grades (for humans) and rewards (for animals) can be detrimental to learning. In particular, focusing on externally-assigned (and somewhat arbitrary) goals, and worrying about whether you are achieving them, does not promote true, deep, and consistent learning.

So we do not do that. Instead, in this course students set personal goals, track their own progress on these goals throughout the semester, amend their goals as needed (setting new goals when previous ones have been achieved; modifying goals that are not realistic), and assessing their your own learning. The course instructors help students achieve their goals for each course component by requesting students to reflect on their progress after each activity, discussing with them challenges that arise as they work toward their individual goals, how to measure and monitor their progress, etc. We will also provide feedback on all course components, to help guide students in improving in any aspect in which they are attempting growth, but without narrowing that feedback to a one-dimensional number or letter. Students’ abilities, challenges, and learning are multidimensional. We have found that providing feedback on these many dimensions is much more useful for promoting growth than are summary grades.

For more information about alternative grading I recommend this excellent episode of Jennifer Gonzalez’s fantastic Cult of Pedagogy, this practical how-to book by Starr Sackstein, and here is a twitter thread where I wrote about this method and discussed it with the community.

Readings & problem sets

  • Assigned reading:
    • Rescorla (1988) – Pavlovian conditioning: It’s not what you think it is
    • Tobler, Dickinson & Schultz (2003) – Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm
    • Bonawitz et al (2011) – The double-edged sword of pedagogy: Instruction limits spontaneous exploration and discovery
  • Supplementary/optional reading:
  • Problem set 1 – Rescorla-Wagner learning
  • Problem set 2 – temporal difference learning
  • Problem set 3 – Q values

Additional resources

  • Gluck, Mercado & Myers – Learning and Memory: From Brain to Behavior – As close to our textbook as there is, though it covers more than we discuss (eg, memory) and does not cover everything that we discuss (eg, computational models)
  • Sutton, R. and Barto, A. – Reinforcement Learning (2018) – A very good and accessible book explaining the computational field of reinforcement learning (also available online)
  • Mackintosh, N.J. – The Psychology of Animal Learning (1974) – Great book, although hard to find (out of print).
  • Mackintosh, N.J. – Conditioning and Associative Learning (1983) – Shorter than the above and covers mostly theoretical aspects, not for complete beginners.
  • Dickinson, A. – Contemporary Animal Learning Theory (1980) – Although this was contemporary a long time ago, it is still very good and easy to read.
  • Classics in the History of Psychology – Classic papers in psychology ONLINE (as well as information about their authors)
  • Behavioral Science Glossary – A useful glossary of many of the terms we use
  • PsychWeb – Resource for psychology links
  • Sniffy the Virtual Rat – 20 day downloadable demo of Sniffy