<abstract xmlns="http://www.w3.org/1999/xhtml"><p>The present paper<sup>1</sup> aims to propose a new type of information-theoretic method to maximize mutual information between inputs and outputs. The importance of mutual information in neural networks is well known, but the actual implementation of mutual information maximization has been quite difficult to undertake. In addition, mutual information has not extensively been used in neural networks, meaning that its applicability is very limited. To overcome the shortcoming of mutual information maximization, we present it here in a very simplified manner by supposing that mutual information is already maximized before learning, or at least at the beginning of learning. The method was applied to three data sets (crab data set, wholesale data set, and human resources data set) and examined in terms of generalization performance and connection weights. The results showed that by disentangling connection weights, maximizing mutual information made it possible to explicitly interpret the relations between inputs and outputs.</p></abstract>

The present paper1 aims to propose a new type of information-theoretic method to maximize mutual information between inputs and outputs. The importance of mutual information in neural networks is well known, but the actual implementation of mutual information maximization has been quite difficult to undertake. In addition, mutual information has not extensively been used in neural networks, meaning that its applicability is very limited. To overcome the shortcoming of mutual information maximization, we present it here in a very simplified manner by supposing that mutual information is already maximized before learning, or at least at the beginning of learning. The method was applied to three data sets (crab data set, wholesale data set, and human resources data set) and examined in terms of generalization performance and connection weights. The results showed that by disentangling connection weights, maximizing mutual information made it possible to explicitly interpret the relations between inputs and outputs.

Supposed Maximum Mutual Information for Improving Generalization and Interpretation of Multi-Layered Neural Networks

Journal of Artificial Intelligence and Soft Computing Research

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

The present paper1 aims to propose a new type of information-theoretic method to maximize mutual information between inputs and outputs. The importance of...

Supposed Maximum Mutual Information for Improving Generalization and Interpretation of Multi-Layered Neural Networks

Publicado en línea: 31 dic 2018

Páginas: 123 - 147

Recibido: 06 feb 2018

Aceptado: 13 ago 2018

DOI: https://doi.org/10.2478/jaiscr-2018-0029

Palabras clave
mutual information, disentanglement, generalization, interpretation

© 2018 Ryotaro Kamimura, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Supposed Maximum Mutual Information for Improving Generalization and Interpretation of Multi-Layered Neural Networks

Publicado en línea: 31 dic 2018

Páginas: 123 - 147

Recibido: 06 feb 2018

Aceptado: 13 ago 2018

DOI: https://doi.org/10.2478/jaiscr-2018-0029

Palabras clavemutual information, disentanglement, generalization, interpretation

© 2018 Ryotaro Kamimura, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Palabras clave
mutual information, disentanglement, generalization, interpretation