<abstract xmlns="http://www.w3.org/1999/xhtml"><sec><h3>Purpose</h3><p>We propose InParTen2, a multi-aspect parallel factor analysis three-dimensional tensor decomposition algorithm based on the Apache Spark framework. The proposed method reduces re-decomposition cost and can handle large tensors.</p></sec><sec><h3>Design/methodology/approach</h3><p>Considering that tensor addition increases the size of a given tensor along all axes, the proposed method decomposes incoming tensors using existing decomposition results without generating sub-tensors. Additionally, InParTen2 avoids the calculation of Khari–Rao products and minimizes shuffling by using the Apache Spark platform.</p></sec><sec><h3>Findings</h3><p>The performance of InParTen2 is evaluated by comparing its execution time and accuracy with those of existing distributed tensor decomposition methods on various datasets. The results confirm that InParTen2 can process large tensors and reduce the re-calculation cost of tensor decomposition. Consequently, the proposed method is faster than existing tensor decomposition algorithms and can significantly reduce re-decomposition cost.</p></sec><sec><h3>Research limitations</h3><p>There are several Hadoop-based distributed tensor decomposition algorithms as well as MATLAB-based decomposition methods. However, the former require longer iteration time, and therefore their execution time cannot be compared with that of Spark-based algorithms, whereas the latter run on a single machine, thus limiting their ability to handle large data.</p></sec><sec><h3>Practical implications</h3><p>The proposed algorithm can reduce re-decomposition cost when tensors are added to a given tensor by decomposing them based on existing decomposition results without re-decomposing the entire tensor.</p></sec><sec><h3>Originality/value</h3><p>The proposed method can handle large tensors and is fast within the limited-memory framework of Apache Spark. Moreover, InParTen2 can handle static as well as incremental tensor decomposition.</p></sec></abstract>

PurposeWe propose InParTen2, a multi-aspect parallel factor analysis three-dimensional tensor decomposition algorithm based on the Apache Spark framework. The proposed method reduces re-decomposition cost and can handle large tensors.Design/methodology/approachConsidering that tensor addition increases the size of a given tensor along all axes, the proposed method decomposes incoming tensors using existing decomposition results without generating sub-tensors. Additionally, InParTen2 avoids the calculation of Khari–Rao products and minimizes shuffling by using the Apache Spark platform.FindingsThe performance of InParTen2 is evaluated by comparing its execution time and accuracy with those of existing distributed tensor decomposition methods on various datasets. The results confirm that InParTen2 can process large tensors and reduce the re-calculation cost of tensor decomposition. Consequently, the proposed method is faster than existing tensor decomposition algorithms and can significantly reduce re-decomposition cost.Research limitationsThere are several Hadoop-based distributed tensor decomposition algorithms as well as MATLAB-based decomposition methods. However, the former require longer iteration time, and therefore their execution time cannot be compared with that of Spark-based algorithms, whereas the latter run on a single machine, thus limiting their ability to handle large data.Practical implicationsThe proposed algorithm can reduce re-decomposition cost when tensors are added to a given tensor by decomposing them based on existing decomposition results without re-decomposing the entire tensor.Originality/valueThe proposed method can handle large tensors and is fast within the limited-memory framework of Apache Spark. Moreover, InParTen2 can handle static as well as incremental tensor decomposition.

Multi-Aspect Incremental Tensor Decomposition Based on Distributed In-Memory Big Data Systems

Department of Computer Science and Engineering

Journal of Data and Information Science

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

PurposeWe propose InParTen2, a multi-aspect parallel factor analysis three-dimensional tensor decomposition algorithm based on the Apache Spark framework. The...

Multi-Aspect Incremental Tensor Decomposition Based on Distributed In-Memory Big Data Systems

Article Category: Research Paper

Publié en ligne: 20 mai 2020

Pages: 13 - 32

Reçu: 30 oct. 2019

Accepté: 06 mars 2020

DOI: https://doi.org/10.2478/jdis-2020-0010

Mots clés
PARAFAC, Tensor decomposition, Incremental tensor decomposition, Apache Spark, Big data

© 2020 Hye-Kyung Yang et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Multi-Aspect Incremental Tensor Decomposition Based on Distributed In-Memory Big Data Systems

Article Category: Research Paper

Publié en ligne: 20 mai 2020

Pages: 13 - 32

Reçu: 30 oct. 2019

Accepté: 06 mars 2020

DOI: https://doi.org/10.2478/jdis-2020-0010

Mots clésPARAFAC, Tensor decomposition, Incremental tensor decomposition, Apache Spark, Big data

© 2020 Hye-Kyung Yang et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Mots clés
PARAFAC, Tensor decomposition, Incremental tensor decomposition, Apache Spark, Big data