Intrinsically motivated learning of real-world sensorimotor skills with developmental constraints

Summary

• Real-world sensorimotor spaces possess three challenging properties: unlearnability, high-dimensionality, and unboundedness
• Argue that exploration in such spaces is constrained by mechanisms in adddition to intrinsic motivation, such as sensorimotor primitives, embodiment, maturational processes, and social guidance
• Perform robot learning experiments that integrate these developmental mechanisms
• Links: [ pdf ]

Background

• Central goal of developmental robotics is to understand that mechanisms that allow for life-long, open-ended learning in the real-world
• A core challenge is task-independent mechanisms for encouraging exploration
  • However, within a life-time, only a small subset of skills can be learned
  • What to explore and what to learn? What not to explore and what not to learn?
• Two broad strategies: internally guided and socially guided exploration
  • In psychology, “interestingness” related to concepts like novelty, reduction of cognitive dissonances, personal causation, and optimal challenge
• Many computation techniques for intrinsically motivated reinforcement learning make assumptions that do not hold in the real-world:
  1. it is possible to learn a complete model of the world
  2. the world is learnable everywhere
  3. the noise is homogenous
• Fundamental properties of real-world developmental environments: Unlearnability, high-dimensionality, and unboundedness
  • Computing meaningful measurs of “interestingness” requires a sampling density that becomes inefficient as dimensionality grows
  • Circular problem occurs in actually evaluating the “interestingness” measure

Methods

• Introduce complementary developmental mechanisms that constrain the growth of the size, dimensionality, and complexity of the exploration space
• Infant development inspired developmental constraints:
  • Sensorimotor primitives: neurally embedded dynamical systems that generate parameterized coordinated movements
  • Task-level intrinsically motivated exploration: while sensorimotor space is high-dimensional and redundant, task space is often relatively low-dimensional
  • Maturational constraints: the degrees of freedom for infants progressively grow
  • Social guidance: bi-directional interaction between social guidance and intrinsic motivation

Results

• Hierarchical RL uses innate temporraly extended skill “templates” called “options”, which enables learning of long action sequences
• SAGG-RIAC integrates task level/goal exploration and control level/joint exploration in single hierarchical active learning architecture
• McSAGG couples SAGG-RIAC with maturational constraints, where the intrinsic motivation system also regulates the bounds of the explorable space
• Social guidance allows an intrinsically motivated learner to discover new explorable task spaces as well as successful control strategies

Conclusion

• Need to figure out how to scale up proof-of-concept initial results to the real world
  • Unlearnability, high-dimensionality, and unboundedness introduce fundamental obstacles
• Intrinsic motivation is a core component, but must be accompanied by complementary mechanisms and integrated into a single developmental architecture