<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><p>Orseau and Ring, as well as Dewey, have recently described problems, including self-delusion, with the behavior of agents using various definitions of utility functions. An agent's utility function is defined in terms of the agent's history of interactions with its environment. This paper argues, via two examples, that the behavior problems can be avoided by formulating the utility function in two steps: 1) inferring a model of the environment from interactions, and 2) computing utility as a function of the environment model. Basing a utility function on a model that the agent must learn implies that the utility function must initially be expressed in terms of specifications to be matched to structures in the learned model. These specifications constitute prior assumptions about the environment so this approach will not work with arbitrary environments. But the approach should work for agents designed by humans to act in the physical world. The paper also addresses the issue of self-modifying agents and shows that if provided with the possibility to modify their utility functions agents will not choose to do so, under some usual assumptions.</p></abstract>

Orseau and Ring, as well as Dewey, have recently described problems, including self-delusion, with the behavior of agents using various definitions of utility functions. An agent's utility function is defined in terms of the agent's history of interactions with its environment. This paper argues, via two examples, that the behavior problems can be avoided by formulating the utility function in two steps: 1) inferring a model of the environment from interactions, and 2) computing utility as a function of the environment model. Basing a utility function on a model that the agent must learn implies that the utility function must initially be expressed in terms of specifications to be matched to structures in the learned model. These specifications constitute prior assumptions about the environment so this approach will not work with arbitrary environments. But the approach should work for agents designed by humans to act in the physical world. The paper also addresses the issue of self-modifying agents and shows that if provided with the possibility to modify their utility functions agents will not choose to do so, under some usual assumptions.

Model-based Utility Functions

Space Science and Engineering Center, University of Wisconsin - Madison, 1225 W. Dayton Street Madison, WI 53706, USA

Journal of Artificial General Intelligence

{"article-title":"Model-based Utility Functions"}

Orseau and Ring, as well as Dewey, have recently described problems, including self-delusion, with the behavior of agents using various definitions of utility...

Model-based Utility Functions

Published Online: May 11, 2012

Page range: 1 - 24

DOI: https://doi.org/10.2478/v10229-011-0013-5

Keywords
rational agent, utility function, self-delusion, self-modification

This content is open access.

Model-based Utility Functions

Published Online: May 11, 2012

Page range: 1 - 24

DOI: https://doi.org/10.2478/v10229-011-0013-5

Keywordsrational agent, utility function, self-delusion, self-modification

This content is open access.

Keywords
rational agent, utility function, self-delusion, self-modification