Scaling Reinforcement Learning to Robotics by Exploiting the Subsumption Architecture

Abstract

Making robots learn complex tasks from reinforcement seems attractive, but a number of problems are encountered in practice. The learning converges slowly because rewards are infrequent, and it is difficult to find effective ways of encoding state history. This paper shows how these problems are overcome by using a subsumption architecture: each module can be given its own simple reward function, and state history information can be easily encoded in a module's applicability predicate. A real robot called OBELIX (see Figure 1) is described that learns several component behaviors in an example task involving pushing boxes. An experimental study demonstrates the feasibility of the subsumption-based approach, and its superiority to a monolithic architecture.