<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">Cognitive Science and Artificial Intelligence share compatible goals of understanding and possibly generating broadly intelligent behavior. In order to determine if progress is made, it is essential to be able to evaluate the behavior of complex computational models, especially those built on general cognitive architectures, and compare it to benchmarks of intelligent behavior such as human performance. Significant methodological challenges arise, however, when trying to extend approaches used to compare model and human performance from tightly controlled laboratory tasks to complex tasks involving more open-ended behavior. This paper describes a model comparison challenge built around a dynamic control task, the Dynamic Stocks and Flows. We present and discuss distinct approaches to evaluating performance and comparing models. Lessons drawn from this challenge are discussed in light of the challenge of using cognitive architectures to achieve Artificial General Intelligence.</abstract>

Psychology Department, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA

Dynamic Decision Making Laboratory, Social and Decision Sciences Department, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA

Alion Science and Technology, 4949 Pearl East Circle, Suite 200, Boulder, CO 80301, USA

Editorial: Cognitive Architectures, Model Comparison and AGI

<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">This paper describes the creation of a cognitive model submitted to the ‘Dynamic Stocks and Flows’ (DSF) modeling challenge. This challenge aims at comparing computational cognitive models for human behavior during an open ended control task. Participants in the modeling competition were provided with a simulation environment and training data for benchmarking their models while the actual specification of the competition task was withheld. To meet this challenge, the cognitive model described here was designed and optimized for generalizability. Only two simple assumptions about human problem solving were used to explain the empirical findings of the training data. In-depth analysis of the data set prior to the development of the model led to the dismissal of correlations or other parametric statistics as goodness-of-fit indicators. A new statistical measurement based on rank orders and sequence matching techniques is being proposed instead. This measurement, when being applied to the human sample, also identifies clusters of subjects that use different strategies for the task. The acceptability of the fits achieved by the model is verified using permutation tests.</abstract>

Keep it simple - A case study of model development in the context of the Dynamic Stocks and Flows (DSF) task

<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">Model comparison is vital to evaluating progress in the fields of artificial general intelligence (AGI) and cognitive architecture. As they mature, AGI and cognitive architectures will become increasingly capable of providing a single model that completes a multitude of tasks, some of which the model was not specifically engineered to perform. These models will be expected to operate for extended periods of time and serve functional roles in real-world contexts. Questions arise regarding how to evaluate such models appropriately, including issues pertaining to model comparison and validation. In this paper, we specifically address model validation across multiple levels of abstraction, using an existing computational process model of unmanned aerial vehicle basic maneuvering to illustrate the relationship between validity and timescales of analysis.</abstract>

Validating Computational Cognitive Process Models across Multiple Timescales

<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">The cognitive modeling and artificial general intelligence research communities may reap greater scientific return on research investments - may achieve an improved understanding of architectures and models - if there is more emphasis on systematic sensitivity and necessity analyses during model development, evaluation, and comparison. We demonstrate this methodological prescription with two of the models submitted for the Dynamic Stocks and Flows (DSF) Model Comparison Challenge, exploring the complex interactions among architectural mechanisms, knowledge-level strategy variants, and task conditions. To cope with the computational demands of these analyses we use a predictive analytics approach similar to regression trees, combined with parallelization on high performance computing clusters, to enable large scale, simultaneous search and exploration.</abstract>

Air Force Research Laboratory, 711 HPW/RHAC, 2698 G Street, Bldg 190, WPAFB, OH 45433, USA

Lockheed Martin at Air Force Research Laboratory, 6030 S. Kent St Mesa, AZ 85212, USA

Department of Psychology, Carnegie Mellon University, 5000 Forbes Ave Pittsburgh, PA 15213, USA

Exploration for Understanding in Cognitive Modeling

<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">This paper describes a model of dynamic decision making in the <italic>Dynamic Stocks and Flows</italic> (DSF) task, developed using the ACT-R cognitive architecture. This task is a simple simulation of a water tank in which the water level must be kept constant whilst the inflow and outflow changes at varying rates. The basic functions of the model are based around three steps. Firstly, the model predicts the water level in the next cycle by adding the current water level to the predicted net inflow of water. Secondly, based on this projection, the net outflow of the water is adjusted to bring the water level back to the target. Thirdly, the predicted net inflow of water is adjusted to improve its accuracy in the future. If the prediction has overestimated net inflow then it is reduced, if it has underestimated net inflow it is increased. The model was entered into a model comparison competition—the <italic>Dynamic Stocks and Flows Challenge</italic>—to model human performance on four conditions of the DSF task and then subject the model to testing on five unseen transfer conditions. The model reproduced the main features of the development data reasonably well but did not reproduce human performance well under the transfer conditions. This suggests that the principles underlying human performance across the different conditions differ considerably despite their apparent similarity. Further lessons for the future development of our model and model comparison challenges are considered.</abstract>

Department of Behavioural and Social Sciences, University of Huddersfield, Queensgate, Huddersfield, HD1 3DH, UK

Department of Psychology, Faculty of Arts and Human Sciences, University of Surrey, Guildford, Surrey, GU2 7XH, UK

Modelling Dynamic Decision Making with the ACT-R Cognitive Architecture

<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">We present a cognitive model performing the Dynamic Stocks&amp;Flows control task, in which subjects control a system by counteracting a systematically changing external variable. The model uses a metacognitive layer that chooses a task strategy drawn from of two classes of strategies: precise calculation and imprecise estimation. The model, formulated within the ACT-R theory, monitors the success of each strategy continuously using instance-based learning and blended retrieval from declarative memory. The model underspecifies other portions of the task strategies, whose timing was determined as unbiased estimate from empirical data. The model's predictions were evaluated on data collected from novel experimental conditions, which did not inform the model's development and included discontinuous and noisy environmental change functions and a control delay. The model as well as the data show sudden changes in subject error and general learning of control; the model also correctly predicted oscillations of plausible magnitude. With its predictions, the model ranked first among the entries to the 2009 Dynamic Stocks&amp;Flows modeling challenge.</abstract>

Metacognition and Multiple Strategies in a Cognitive Model of Online Control

<abstract xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">The equivalence test (Stewart and West, 2007; Stewart, 2007) is a statistical measure for evaluating the similarity between a model and the system being modelled. It is designed to avoid over-fitting and to generate an easily interpretable summary of the quality of a model. We apply the equivalence test to two tasks: Repeated Binary Choice (Erev et al., 2010) and Dynamic Stocks and Flows (Gonzalez and Dutt, 2007). In the first case, we find a broad range of statistically equivalent models (and win a prediction competition) while identifying particular aspects of the task that are not yet adequately captured. In the second case, we re-evaluate results from the Dynamic Stocks and Flows challenge, demonstrating how our method emphasizes the breadth of coverage of a model and how it can be used for comparing different models. We argue that the explanatory power of models hinges on numerical similarity to empirical data over a broad set of measures.</abstract>

Centre for Theoretical Neuroscience, University of Waterloo, 200 University Avenue West Waterloo, ON N2L 3G1, Canada

Institute of Cognitive Science, Carleton University, 1125 Colonel By Drive Ottawa, ON K1S 5B6, Canada

Testing for Equivalence: A Methodology for Computational Cognitive Modelling

AHEAD OF PRINT

Volume 14 (2023): Issue 1 (March 2023)

Volume 13 (2022): Issue 1 (October 2022)

Volume 12 (2021): Issue 1 (January 2021)

Volume 11 (2020): Issue 2 (February 2020) Special Issue “On Defining Artificial Intelligence” — Commentaries and Author’s Response

Volume 11 (2020): Issue 1 (January 2020)

Volume 10 (2019): Issue 2 (January 2019)

Volume 10 (2019): Issue 1 (January 2019)

Volume 9 (2018): Issue 1 (March 2018)

Volume 8 (2017): Issue 1 (December 2017)

Volume 7 (2016): Issue 1 (December 2016)

Volume 6 (2015): Issue 1 (December 2015)

Volume 5 (2014): Issue 1 (December 2014)

Volume 4 (2013): Issue 3 (December 2013) Brain Emulation and Connectomics: a Convergence of Neuroscience and Artificial General Intelligence, Editors: Randal Koene and Diana Deca

Volume 4 (2013): Issue 2 (December 2013) Conceptual Commitments of AGI Systems, Editors:	Haris Dindo, James Marshall, and Giovanni Pezzulo

Volume 4 (2013): Issue 1 (November 2013)

Volume 3 (2012): Issue 3 (December 2012) Self-Programming and Constructivist Methodologies for AGI,
 Editors: Kristinn R. Thórisson, Eric Nivel and Ricardo Sanz

Volume 3 (2012): Issue 2 (November 2012)

Volume 3 (2012): Issue 1 (May 2012)

Volume 2 (2010): Issue 2 (December 2010) Cognitive Architectures, Model Comparison, and AGI, Editors: Christian Lebiere, Cleotilde Gonzalez and Walter Warwick

Volume 2 (2010): Issue 1 (June 2010)

Volume 1 (2009): Issue 1 (December 2009)

Journal of Artificial General Intelligence

 Journal of Artificial General Intelligence (JAGI) is a peer-reviewed open-access academic journal, owned by the Artificial General Intelligence Society (AGIS).  Artificial General Intelligence (AGI) is an emerging field aiming at the building of "thinking machines", that is, general-purpose systems with intelligence comparable to that of the human mind. While this was the original goal of Artificial Intelligence (AI), the mainstream of AI research has turned toward domain-dependent and problem-specific solutions; therefore it has become necessary to use a new name to indicate research that still pursues the "Grand AI Dream". Similar labels for this kind of research include "Strong AI", "Human-level AI", etc.  The problems involved in creating general-purpose intelligent systems are very different from those involved in creating special-purpose systems. Therefore, this journal is different from conventional AI journals in its stress on the long-term potential of research towards the ultimate goal of AGI, rather than immediate applications. Articles focused on details of AGI systems are welcome, if they clearly indicate the relation between the special topics considered and intelligence as a whole, by addressing the generality, extensibility, and scalability of the techniques proposed or discussed.  Since AGI research is still in its early stage, the journal strongly encourages novel approaches coming from various theoretical and technical traditions, including (but not limited to) symbolic, connectionist, statistical, evolutionary, robotic and information-theoretic, as well as integrative and hybrid approaches.  Rejection Rate  &gt;30%   Archiving  Sciendo archives the contents of this journal in Portico - digital long-term preservation service of scholarly books, journals and collections.  Plagiarism Policy  The editorial board is participating in a growing community of Similarity Check System's users in order to ensure that the content published is original and trustworthy. Similarity Check is a medium that allows for comprehensive manuscripts screening, aimed to eliminate plagiarism and provide a high standard and quality peer-review process. 

Volume 2 (2010): Issue 2 (December 2010) - Cognitive Architectures, Model Comparison, and AGI, Editors: Christian Lebiere, Cleotilde Gonzalez and Walter Warwick

Journal of Artificial General Intelligence

Volume 2 (2010): Issue 2 (December 2010)

Cognitive Architectures, Model Comparison, and AGI, Editors: Christian Lebiere, Cleotilde Gonzalez and Walter Warwick