<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>Residual deep neural networks (ResNets) are mathematically described as interacting particle systems. In the case of infinitely many layers the ResNet leads to a system of coupled system of ordinary differential equations known as neural differential equations. For large scale input data we derive a mean–field limit and show well–posedness of the resulting description. Further, we analyze the existence of solutions to the training process by using both a controllability and an optimal control point of view. Numerical investigations based on the solution of a formal optimality system illustrate the theoretical findings.</p>
</abstract>

Residual deep neural networks (ResNets) are mathematically described as interacting particle systems. In the case of infinitely many layers the ResNet leads to a system of coupled system of ordinary differential equations known as neural differential equations. For large scale input data we derive a mean–field limit and show well–posedness of the resulting description. Further, we analyze the existence of solutions to the training process by using both a controllability and an optimal control point of view. Numerical investigations based on the solution of a formal optimality system illustrate the theoretical findings.

Continuous limits of residual neural networks in case of large input data

“G. Castelnuovo” – Sapienza University of Rome

Communications in Applied and Industrial Mathematics

This work is licensed under the Creative Commons Attribution 4.0 International License.

{"article-title":"Continuous limits of residual neural networks in case of large input data"}

Residual deep neural networks (ResNets) are mathematically described as interacting particle systems. In the case of infinitely many layers the ResNet leads to...

Continuous limits of residual neural networks in case of large input data

Published Online: Dec 24, 2022

Page range: 96 - 120

Received: Jul 11, 2022

Accepted: Nov 12, 2022

DOI: https://doi.org/10.2478/caim-2022-0008

Keywords
Neural networks, mean-field limit, well-posedness, optimal control, controllability

© 2022 Michael Herty et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Continuous limits of residual neural networks in case of large input data

Published Online: Dec 24, 2022

Page range: 96 - 120

Received: Jul 11, 2022

Accepted: Nov 12, 2022

DOI: https://doi.org/10.2478/caim-2022-0008

KeywordsNeural networks, mean-field limit, well-posedness, optimal control, controllability

© 2022 Michael Herty et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Keywords
Neural networks, mean-field limit, well-posedness, optimal control, controllability