<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><p>General-purpose, intelligent, learning agents cycle through sequences of observations, actions, and rewards that are complex, uncertain, unknown, and non-Markovian. On the other hand, reinforcement learning is well-developed for small finite state Markov decision processes (MDPs). Up to now, extracting the right state representations out of bare observations, that is, reducing the general agent setup to the MDP framework, is an art that involves significant effort by designers. The primary goal of this work is to automate the reduction process and thereby significantly expand the scope of many existing reinforcement learning algorithms and the agents that employ them. Before we can think of mechanizing this search for suitable MDPs, we need a formal objective criterion. The main contribution of this article is to develop such a criterion. I also integrate the various parts into one learning algorithm. Extensions to more realistic dynamic Bayesian networks are developed in Part II (Hutter, 2009c). The role of POMDPs is also considered there.</p></abstract>

General-purpose, intelligent, learning agents cycle through sequences of observations, actions, and rewards that are complex, uncertain, unknown, and non-Markovian. On the other hand, reinforcement learning is well-developed for small finite state Markov decision processes (MDPs). Up to now, extracting the right state representations out of bare observations, that is, reducing the general agent setup to the MDP framework, is an art that involves significant effort by designers. The primary goal of this work is to automate the reduction process and thereby significantly expand the scope of many existing reinforcement learning algorithms and the agents that employ them. Before we can think of mechanizing this search for suitable MDPs, we need a formal objective criterion. The main contribution of this article is to develop such a criterion. I also integrate the various parts into one learning algorithm. Extensions to more realistic dynamic Bayesian networks are developed in Part II (Hutter, 2009c). The role of POMDPs is also considered there.

Feature Reinforcement Learning: Part I. Unstructured MDPs

RSISE @ ANU and SML @ NICTA, Canberra, ACT, 0200, Australia

Journal of Artificial General Intelligence

{"article-title":"Feature Reinforcement Learning: Part I. Unstructured MDPs"}

General-purpose, intelligent, learning agents cycle through sequences of observations, actions, and rewards that are complex, uncertain, unknown, and...

Feature Reinforcement Learning: Part I. Unstructured MDPs

Published Online: Nov 23, 2011

Page range: 3 - 24

DOI: https://doi.org/10.2478/v10229-011-0002-8

Keywords
Reinforcement learning, Markov decision process, partial observability, feature learning, explore-exploit, information & complexity, rational agents

This content is open access.

Feature Reinforcement Learning: Part I. Unstructured MDPs

Published Online: Nov 23, 2011

Page range: 3 - 24

DOI: https://doi.org/10.2478/v10229-011-0002-8

KeywordsReinforcement learning, Markov decision process, partial observability, feature learning, explore-exploit, information & complexity, rational agents

This content is open access.

Keywords
Reinforcement learning, Markov decision process, partial observability, feature learning, explore-exploit, information & complexity, rational agents