<abstract xmlns="http://www.w3.org/1999/xhtml">

<sec><h3>Research background</h3>
<p>Markov Decision Processes (<italic>MDPs</italic>) are a powerful framework for modeling many real-world problems with finite-horizons that maximize the reward given a sequence of actions. Although many problems such as investment and financial market problems where the value of a reward decreases exponentially with time, require the introduction of interest rates.</p>
</sec>
<sec><h3>Purpose</h3>
<p>This study investigates non-stationary finite-horizon <italic>MDPs</italic> with a discount factor to account for fluctuations in rewards over time.</p>
</sec>
<sec><h3>Research methodology</h3>
<p>To consider the fluctuations of rewards with time, the authors define new nonstationary finite-horizon <italic>MDPs</italic> with a discount factor. First, the existence of an optimal policy for the proposed finite-horizon discounted <italic>MDPs is proven</italic>. Next, a new Discounted Backward Induction (<italic>DBI</italic>) algorithm is presented to find it. To enhance the value of their proposal, a financial model is used as an example of a finite-horizon discounted <italic>MDP</italic> and an adaptive <italic>DBI</italic> algorithm is used to solve it.</p>
</sec>
<sec><h3>Results</h3>
<p>The proposed method calculates the optimal values of the investment to maximize its expected total return with consideration of the time value of money.</p>
</sec>
<sec><h3>Novelty</h3>
<p>No existing studies have before examined dynamic finite-horizon problems that account for temporal fluctuations in rewards.</p>
</sec>
</abstract>

Research background
Markov Decision Processes (MDPs) are a powerful framework for modeling many real-world problems with finite-horizons that maximize the reward given a sequence of actions. Although many problems such as investment and financial market problems where the value of a reward decreases exponentially with time, require the introduction of interest rates.

Purpose
This study investigates non-stationary finite-horizon MDPs with a discount factor to account for fluctuations in rewards over time.

Research methodology
To consider the fluctuations of rewards with time, the authors define new nonstationary finite-horizon MDPs with a discount factor. First, the existence of an optimal policy for the proposed finite-horizon discounted MDPs is proven. Next, a new Discounted Backward Induction (DBI) algorithm is presented to find it. To enhance the value of their proposal, a financial model is used as an example of a finite-horizon discounted MDP and an adaptive DBI algorithm is used to solve it.

Results
The proposed method calculates the optimal values of the investment to maximize its expected total return with consideration of the time value of money.

Novelty
No existing studies have before examined dynamic finite-horizon problems that account for temporal fluctuations in rewards.

Solving Finite-Horizon Discounted Non-Stationary MDPS

Laboratory of Information Processing and Decision Support

Folia Oeconomica Stetinensia

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.

{"article-title":"Solving Finite-Horizon Discounted Non-Stationary MDPS"}

Research background
Markov Decision Processes (MDPs) are a powerful framework for modeling many real-world problems with finite-horizons that maximize the...

Solving Finite-Horizon Discounted Non-Stationary MDPS

Published Online: Jun 09, 2023

Page range: 1 - 15

Accepted: Feb 26, 2023

DOI: https://doi.org/10.2478/foli-2023-0001

Keywords
Markov Decision Process, Dynamic Programming, Backward Induction algorithm

© 2023 El Akraoui Bouchra et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.

Solving Finite-Horizon Discounted Non-Stationary MDPS

Published Online: Jun 09, 2023

Page range: 1 - 15

Accepted: Feb 26, 2023

DOI: https://doi.org/10.2478/foli-2023-0001

KeywordsMarkov Decision Process, Dynamic Programming, Backward Induction algorithm

© 2023 El Akraoui Bouchra et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.

Keywords
Markov Decision Process, Dynamic Programming, Backward Induction algorithm