<abstract xmlns="http://www.w3.org/1999/xhtml"><p>A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.</p></abstract>

A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.

Encouraging an appropriate representation simplifies training of neural networks

Acta Universitatis Sapientiae, Informatica

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

{"article-title":"Encouraging an appropriate representation simplifies training of neural networks"}

A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this...

Encouraging an appropriate representation simplifies training of neural networks

Publicado en línea: 16 jul 2020

Páginas: 102 - 111

Recibido: 26 mar 2020

Aceptado: 07 jun 2020

DOI: https://doi.org/10.2478/ausi-2020-0007

Palabras clave
neural networks, representation, integration of domain knowledge

© 2020 Krisztian Buza, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Encouraging an appropriate representation simplifies training of neural networks

Publicado en línea: 16 jul 2020

Páginas: 102 - 111

Recibido: 26 mar 2020

Aceptado: 07 jun 2020

DOI: https://doi.org/10.2478/ausi-2020-0007

Palabras claveneural networks, representation, integration of domain knowledge

© 2020 Krisztian Buza, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Palabras clave
neural networks, representation, integration of domain knowledge