<abstract xmlns="http://www.w3.org/1999/xhtml"><p>A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.</p></abstract>

A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.

Encouraging an appropriate representation simplifies training of neural networks

Acta Universitatis Sapientiae, Informatica

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

{"article-title":"Encouraging an appropriate representation simplifies training of neural networks"}

A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this...

Encouraging an appropriate representation simplifies training of neural networks

Data publikacji: 16 lip 2020

Zakres stron: 102 - 111

Otrzymano: 26 mar 2020

Przyjęty: 07 cze 2020

DOI: https://doi.org/10.2478/ausi-2020-0007

Słowa kluczowe
neural networks, representation, integration of domain knowledge

© 2020 Krisztian Buza, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Encouraging an appropriate representation simplifies training of neural networks

Data publikacji: 16 lip 2020

Zakres stron: 102 - 111

Otrzymano: 26 mar 2020

Przyjęty: 07 cze 2020

DOI: https://doi.org/10.2478/ausi-2020-0007

Słowa kluczoweneural networks, representation, integration of domain knowledge

© 2020 Krisztian Buza, published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Słowa kluczowe
neural networks, representation, integration of domain knowledge