Intrinsically motivated learning of real-world sensorimotor skills with developmental constraints

Oudeyer et al., 2013


  • Real-world sensorimotor spaces possess three challenging properties: unlearnability, high-dimensionality, and unboundedness
  • Argue that exploration in such spaces is constrained by mechanisms in adddition to intrinsic motivation, such as sensorimotor primitives, embodiment, maturational processes, and social guidance
  • Perform robot learning experiments that integrate these developmental mechanisms
  • Links: [ pdf ]


  • Central goal of developmental robotics is to understand that mechanisms that allow for life-long, open-ended learning in the real-world
  • A core challenge is task-independent mechanisms for encouraging exploration
    • However, within a life-time, only a small subset of skills can be learned
    • What to explore and what to learn? What not to explore and what not to learn?
  • Two broad strategies: internally guided and socially guided exploration
    • In psychology, “interestingness” related to concepts like novelty, reduction of cognitive dissonances, personal causation, and optimal challenge
  • Many computation techniques for intrinsically motivated reinforcement learning make assumptions that do not hold in the real-world:
    1. it is possible to learn a complete model of the world
    2. the world is learnable everywhere
    3. the noise is homogenous
  • Fundamental properties of real-world developmental environments: Unlearnability, high-dimensionality, and unboundedness
    • Computing meaningful measurs of “interestingness” requires a sampling density that becomes inefficient as dimensionality grows
    • Circular problem occurs in actually evaluating the “interestingness” measure


  • Introduce complementary developmental mechanisms that constrain the growth of the size, dimensionality, and complexity of the exploration space
  • Infant development inspired developmental constraints:
    • Sensorimotor primitives: neurally embedded dynamical systems that generate parameterized coordinated movements
    • Task-level intrinsically motivated exploration: while sensorimotor space is high-dimensional and redundant, task space is often relatively low-dimensional
    • Maturational constraints: the degrees of freedom for infants progressively grow
    • Social guidance: bi-directional interaction between social guidance and intrinsic motivation


  • Hierarchical RL uses innate temporraly extended skill “templates” called “options”, which enables learning of long action sequences
  • SAGG-RIAC integrates task level/goal exploration and control level/joint exploration in single hierarchical active learning architecture
  • McSAGG couples SAGG-RIAC with maturational constraints, where the intrinsic motivation system also regulates the bounds of the explorable space
  • Social guidance allows an intrinsically motivated learner to discover new explorable task spaces as well as successful control strategies


  • Need to figure out how to scale up proof-of-concept initial results to the real world
    • Unlearnability, high-dimensionality, and unboundedness introduce fundamental obstacles
  • Intrinsic motivation is a core component, but must be accompanied by complementary mechanisms and integrated into a single developmental architecture
Elias Z. Wang
Elias Z. Wang
AI Researcher | PhD Candidate