<abstract xmlns="http://www.w3.org/1999/xhtml"><p>A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.</p></abstract>

A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.

Encouraging an appropriate representation simplifies training of neural networks

Acta Universitatis Sapientiae, Informatica

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

{"article-title":"Encouraging an appropriate representation simplifies training of neural networks"}

A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this...

Encouraging an appropriate representation simplifies training of neural networks

Pubblicato online: 16 lug 2020

Pagine: 102 - 111

Ricevuto: 26 mar 2020

Accettato: 07 giu 2020

DOI: https://doi.org/10.2478/ausi-2020-0007

Parole chiave
neural networks, representation, integration of domain knowledge

© 2020 Krisztian Buza, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Encouraging an appropriate representation simplifies training of neural networks

Pubblicato online: 16 lug 2020

Pagine: 102 - 111

Ricevuto: 26 mar 2020

Accettato: 07 giu 2020

DOI: https://doi.org/10.2478/ausi-2020-0007

Parole chiaveneural networks, representation, integration of domain knowledge

© 2020 Krisztian Buza, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Parole chiave
neural networks, representation, integration of domain knowledge