<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>The Transformer is an important addition to the rapidly increasing list of different Artificial Neural Networks (ANNs) suited for extremely complex automation tasks. It has already gained the position of the tool of choice in automatic translation in many business solutions. In this paper, we present an automated approach to optimizing the Transformer structure based upon Simulated Annealing, an algorithm widely recognized for both its simplicity and usability in optimization tasks where the search space may be highly complex. The proposed method allows for the use of parallel computing and time-efficient optimization, thanks to modifying the structure while training the network rather than performing the two one after another. The algorithm presented does not reset the weights after changes in the transformer structure. Instead, it continues the training process to allow the results to be adapted without randomizing all the training parameters. The algorithm has shown a promising performance during experiments compared to traditional training methods without structural modifications. The solution has been released as open-source to facilitate further development and use by the machine learning community.</p>
</abstract>

The Transformer is an important addition to the rapidly increasing list of different Artificial Neural Networks (ANNs) suited for extremely complex automation tasks. It has already gained the position of the tool of choice in automatic translation in many business solutions. In this paper, we present an automated approach to optimizing the Transformer structure based upon Simulated Annealing, an algorithm widely recognized for both its simplicity and usability in optimization tasks where the search space may be highly complex. The proposed method allows for the use of parallel computing and time-efficient optimization, thanks to modifying the structure while training the network rather than performing the two one after another. The algorithm presented does not reset the weights after changes in the transformer structure. Instead, it continues the training process to allow the results to be adapted without randomizing all the training parameters. The algorithm has shown a promising performance during experiments compared to traditional training methods without structural modifications. The solution has been released as open-source to facilitate further development and use by the machine learning community.

Optimizing the Structures of Transformer Neural Networks Using Parallel Simulated Annealing

Journal of Artificial Intelligence and Soft Computing Research

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

The Transformer is an important addition to the rapidly increasing list of different Artificial Neural Networks (ANNs) suited for extremely complex automation...

Optimizing the Structures of Transformer Neural Networks Using Parallel Simulated Annealing

Publicado en línea: 11 jun 2024

Páginas: 267 - 282

Recibido: 09 ene 2024

Aceptado: 26 may 2024

DOI: https://doi.org/10.2478/jaiscr-2024-0015

Palabras clave
neural architecture search, transformers, natural language processing, simulated annealing

© 2024 Maciej Trzciński et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Optimizing the Structures of Transformer Neural Networks Using Parallel Simulated Annealing

Publicado en línea: 11 jun 2024

Páginas: 267 - 282

Recibido: 09 ene 2024

Aceptado: 26 may 2024

DOI: https://doi.org/10.2478/jaiscr-2024-0015

Palabras claveneural architecture search, transformers, natural language processing, simulated annealing

© 2024 Maciej Trzciński et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Palabras clave
neural architecture search, transformers, natural language processing, simulated annealing