<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>The Feature Markov Decision Processes ( MDPs) model developed in Part I (Hutter, 2009b) is well-suited for learning agents in general environments. Nevertheless, unstructured (Φ)MDPs are limited to relatively simple environments. Structured MDPs like Dynamic Bayesian Networks (DBNs) are used for large-scale real-world problems. In this article I extend ΦMDP to ΦDBN. The primary contribution is to derive a cost criterion that allows to automatically extract the most relevant features from the environment, leading to the “best” DBN representation. I discuss all building blocks required for a complete general learning algorithm, and compare the novel ΦDBN model to the prevalent POMDP approach.</p>
</abstract>

The Feature Markov Decision Processes ( MDPs) model developed in Part I (Hutter, 2009b) is well-suited for learning agents in general environments. Nevertheless, unstructured (Φ)MDPs are limited to relatively simple environments. Structured MDPs like Dynamic Bayesian Networks (DBNs) are used for large-scale real-world problems. In this article I extend ΦMDP to ΦDBN. The primary contribution is to derive a cost criterion that allows to automatically extract the most relevant features from the environment, leading to the “best” DBN representation. I discuss all building blocks required for a complete general learning algorithm, and compare the novel ΦDBN model to the prevalent POMDP approach.

Feature Reinforcement Learning: Part II. Structured MDPs

Journal of Artificial General Intelligence

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

{"article-title":"Feature Reinforcement Learning: Part II. Structured MDPs"}

The Feature Markov Decision Processes ( MDPs) model developed in Part I (Hutter, 2009b) is well-suited for learning agents in general environments....

Feature Reinforcement Learning: Part II. Structured MDPs

Published Online: Jun 14, 2021

Page range: 71 - 86

Received: Oct 21, 2020

Accepted: Apr 06, 2021

DOI: https://doi.org/10.2478/jagi-2021-0003

Keywords
Reinforcement learning, dynamic Bayesian network, structure learning, feature selection, global vs. local reward, explore-exploit, information & complexity, rational agents, partial observability

© 2021 Marcus Hutter, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Feature Reinforcement Learning: Part II. Structured MDPs

Published Online: Jun 14, 2021

Page range: 71 - 86

Received: Oct 21, 2020

Accepted: Apr 06, 2021

DOI: https://doi.org/10.2478/jagi-2021-0003

KeywordsReinforcement learning, dynamic Bayesian network, structure learning, feature selection, global vs. local reward, explore-exploit, information & complexity, rational agents, partial observability

© 2021 Marcus Hutter, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Keywords
Reinforcement learning, dynamic Bayesian network, structure learning, feature selection, global vs. local reward, explore-exploit, information & complexity, rational agents, partial observability