A long command subsequence algorithm for manufacturing industry recommendation systems with similarity connection technology

With the rapid development and popularisation of internet technology, a huge number of information resources appear on the internet, as to which users find it difficult to achieve or attain the specific information they are in need of or searching for. Users are not successful in finding information resources who are at time crunch, and many of them cannot interpret if the resource found by them is correct, which wastes a lot of their time and information resources. At the same time, users have to invest much time and energy to search for information resources they need. Such a phenomenon with booming information resources but low utilisation rates of information resources is known as ‘information overload’. This problem can be traditionally solved by using search engines, but search engines do not take users’ personalised demands into consideration. Though different people search for information using different keywords, they only achieve the same results. Therefore, the problem of information overload is still not well solved. In order to solve this problem, this paper puts forward personalised service technology which aims to provide different information resources for different users. By counting on users’ own information, this technology can find differences between users and their own characteristics. Besides, it can provide different users with targeted, differentiated services according to the above features.

At present, many scholars have done a lot of researches on personalised services whose core point is the personalised recommendation system. Therefore, the recommendation system is an important part of the research work of personalised service technology. Recommendation algorithm plays the most important role in the recommendation system [1], and the advantages and disadvantages of the algorithm affect the recommendation performance of the system.

Some existing recommendation systems can be divided as follows: recommendation system [2] based on knowledge contents, collaborative filtering recommendation system and mixed recommendation system. Many manufacturing industry systems have fully appreciated the benefits of the recommendation system, but the recommendation system is also facing severe security problems. Due to the openness of the recommendation system itself and its sensitivity to user information, people begin to pay more and more attention to the trusted recommendation mechanism of the collaborative filtration recommendation system, making it gradually a hot research topic.

Similarity connection is used to find all data groups whose similar values are greater than the threshold, which are given by users from a given data set under the specified similarity function measure. Similarity connection has been widely used in many fields. For example, in geography, similarity connection can be used to detect the collision or proximity of geographical features, such as landmarks, houses and roads. In the application of medical imaging, similarity connection can be used to detect whether the direct distances of some cancer cells are less than a certain threshold. In the recommendation system, similarity connection can be used to compare the similarity of recommended items. However, if the similarity connection is used for unordered and unindexed data sets, it will cost a lot in computation.

Because the scale of data grew very fast in the recent years, similarity connection is a bottleneck which blocks its development to a larger scale. The enquiry done through similarity connection of mass data means that all groups of similar objects can be searched from mass data sources. This is a basic operation in similarity connection's dealing with mass data as it attracts wide attention.

Similarity connection query of data can be divided into the following three main categories according to different classification standards: (1) in terms of different definitions of “similarity”, it can be divided into threshold join query and Top-k join query; (2) in terms of difference in the number of data sources, it can be divided into single-source (self-join) query, double-source query and multi-source join query (single-source join query means that all objects of similar two-tuples found come from the same data source) and (3) in terms of different data types, it can be divided into set, character string, vector, graph and other join queries. Join queries of data sets aim to deal with data sets. Similarity connection queries of different categories, such as threshold connection query, can be interlocked. It can be a single-source connection query or dual-source connection query.

There are still many problems on new characteristics of data development and application development that have not been studied yet, for example, (1) high-dimensional data similarity connection technology and (2) massive online real-time similarity connection technology. Similarity connection query of high-dimensional data faces great challenges due to the existence of dimension disasters. The traditional query algorithm with index structure as the basis does not work now. Similarity connection queries of high-dimensional data will be basic operations of many data mining and machine learning tasks. For massive online real-time similarity connection technology, at present, the Map-Reduce framework featured by good extensibility, fault tolerance and usability is used to perform the similarity connection of mass data. However, as a batch processing model, Map-Reduce is not suitable for real-time data processing. The online real-time processing of similarity connection of mass data needs to be further studied. In order to solve these two problems, we should think about (1) whether we can use the piece-wise cumulative approximation method to reduce the high-dimensional data to a low-dimensional space, which would help us overcome the shortcoming of the rapid performance decline caused by the increase of dimensions and carry out effective filtering at a lower cost, and whether the parallel join query algorithm can be used to process large-scale high-dimensional data? (2) Whether data processed through dimension reduction filtration can be further divided into several subspaces according to certain rules and be prepared for streaming data processing in memory in the next step? (3) Can the reverse index list of character strings be regarded as states to iteratively process the incremental character string of streaming data? Can the new states be used to carry out similarity connection for the incremental data and to process online data in real time rapidly within limited memories?

y calculating the unilateral function of multiple sets and their conjunctive functions. In order to calculate the unilateral function, we only need to A lot of innovative achievements have been made in terms of similarity connection query technology. However, there is no unified definition of similarity connection query in domestic and foreign literature. For example, Shim et al. [4] define spatial similarity connection query as follows: (1) self-connection (when given a set consisting of high-dimensional points and a distance metric, you should find all point pairs whose distances do not exceed the threshold from this set) and (2) non-self-connection (when given two sets of points consisting of higher dimensional points and a distance metric, you should find all pairs of points that meet the following conditions: two points come from each of these sets and the distance does not exceed the threshold). Xiao et al. [5] defined Top-k similarity connection query as follows: when given two sets and similarity functions composed of records, you should find the most similar record pairs from these two sets.

There are also some research results on the similarity connection of massive data. Ma et al. [6] put forward a method based on Map-Reduce and filtering technology, that is, set Jaccard-Similarity coefficient is used to complete single-source or double-source record join query based on set similarity. In the paper, the Map-Reduce framework is used for the first time to solve the similarity connection query problem of massive set data and the following two operations are done to improve the algorithm efficiency: (1) Prefix filtering technology is used in Map phase and length, position and suffix filtering technologies are used in Reduce phase, which reduces the number of copies and computing times. (2) All collection elements are sorted in ascending order according to word frequency, which can not only enhance the effect of prefix filtering but also ensure the load balance to a certain extent. Rafiei and Deng [7] proposed several general algorithms based on Map-Reduce. That is to say, the said Jaccard-Similarity coefficient, hamming distance function or character string editing distance function are used to complete the similarity connection queries of the single-source collection, bit character string or character string. Original data are divided into several groups by means of replicas; then, some results obtained in each group will be processed in parallel, and all results will be aggregated to get the final result. At the same time, there shall be as few copies as possible and shall be no duplicate calculation and no duplicate output results. Vernica et al. [8] proposed an iterative division method based on Map-Reduce and completed the similarity connection query of single-source or double-source metric space

This paper carries out iterative division of original data sets so that the data sets obtained can be as small as possible. Then, an efficient memory similarity connection algorithm is used to process them so as to obtain partial results. Finally, partial results are summarised to obtain the final result. Albeanu [9] put forward that a two-step method based on Map-Reduce and filtering technology shall be used to complete the similarity connection query of single-source multiple collections, collections, character strings or vectors. The basic idea is that the similarity calculation of multiple set pairs can be obtained by scan multiple sets, and in order to calculate conjunctive function, we need to scan the intersection. Therefore, we need to calculate the unilateral function of multiple sets and their conjunctive function before we calculate their similarity and obtain the final result. Silva and Reed [10] put forward the Map-Reduce algorithm, which uses Euclidean distance function to complete the query of single-source vector Top-k similarity connection.

The fundamental concept is as follows: copies shall be used to divide the original data sets into different groups. The Top-k similar vector pairs of each group will be then counted. After that, the results of all groups shall be aggregated and ranked to obtain top-k similar vector pairs of the original data sets.

This paper adopts a space efficient algorithm for the longest common subsequence (LCS) in k-length substrings. The LCS problem is a classic problem in computer science. Given two sequences A and B, the LCS problem is to find subsequence of A and B whose length is the longest among all common subsequences of the two given sequences. The problem has numerous applications in many apparently unrelated fields ranging from file comparison, pattern matching, computational biology, etc. [12].

By adopting the optimised incremental similarity connection method based on memory computation optimised above to process streaming data, we can obtain the longest common additive sequence of two given input sequences easily. On the basis of the recursive equation, this paper uses a very simple linear space algorithm to solve this problem and adopts a new state to carry out similarity connection of incremental data.

The experiment uses the fabric data sets collected in large textile factories, and such data can be used for fabric recommendation. There are 972 kinds of classification in the data set, 486 kinds of training set are divided semi randomly and the remaining 486 kinds are used as the test set. This paper compared the content-based recommendation method BMR, the recommendation method BFR based on collaborative filtering, the improved similarity connection technology recommendation method TBRR [13] and the LCS similarity connection technology recommendation method LSCR proposed in this paper.

Besides, the fabric data sets also compared the three indexes of recall rate (RECALL), precision (PRECISION) and mean average error (MAE); these are the recommended systems to measure the accuracy of the forecast.

Eq. (6) is the recalculation method of RECALL. (6) RECALL=∑u∈U| R(u)∩T(u) |∑u∈U| T(u) | RECALL = {{\sum\limits_{u \in U} \left| {R(u) \cap T(u)} \right|} \over {\sum\limits_{u \in U} \left| {T(u)} \right|}} Eq. (7) is the calculation method of PRECISION. (7) PRECISION=∑u∈U| R(u)∩T(u) |∑u∈U| R(u) | PRECISION = {{\sum\limits_{u \in U} \left| {R(u) \cap T(u)} \right|} \over {\sum\limits_{u \in U} \left| {R(u)} \right|}} Eq. (8) is the calculation method of F1-MEASURE. (8) F1=2×RECALL×PRECISIONRECALL+RPECISION F1 = 2 \times {{RECALL \times PRECISION} \over {RECALL + RPECISION}} Here, R(u) denotes the recommended list of user u and T (u) denotes the list of behaviour records of user u in the test set, and the larger the values of these two indexes are, the better the algorithm is.

Eq. (9) is the computation method of MAE (9) MAE=∑i=1N| pi−qi |N MAE = {{\sum\limits_{i = 1}^N \left| {{p_i} - {q_i}} \right|} \over N} Here, p_i is the predicted score set, N is the predicted number of items and q_i is the actual score set, which reflects the recommended accuracy by calculating the deviation between the predicted and actual values. The smaller the MAE value is, the closer the score is to the true score, the more accurate the forecast is and the better the final recommendation is.

As shown in Figures 2–5, we compare the content-based recommendation method BMR, the collaborative filtering-based recommendation method BFR, the time-related composite filtering recommendation TBRR [13] and the recall (RECALL), precision (PRECISION), f1(F1-MEASURE) and mean absolute error (MAE) of the LCS similarity connection technology recommendation method LSCR proposed by us in the fabric data sets with recommended values of k = 1, 5, 10, 40, 50.

It can be seen that the LSCR proposed in this paper combines the advantages of the optimised incremental similarity connection method and the longest common additive sequence of two given input sequences, and the related technical indicators are superior to the traditional BMR, BFR and our previous algorithm TBRR.

Fig. 2

Fig. 3

Fig. 4

Fig. 5

In order to solve the timeliness problem of the recommendation system, this paper extracts time series of data according to the time distribution of data and adopts incremental processing method to greatly reduce the calculation amount on the premise that the precision of real-time recommendation system is ensured. For fabric data sets, the technology used in this paper is better than the traditional BMR, BFR and TBRR algorithms in recall, precision and MAE. The characteristic of this paper is to solve the dimensionality reduction method in the process of processing massive similarity connection technology and the method of using the longest common substring algorithm in memory computing. The time series are cut according to the piecewise accumulation in the time series, and the incremental processing method is adopted to greatly reduce the amount of calculation under the condition of ensuring the accuracy of the real-time recommendation system. The optimised incremental similarity connection method based on memory computing is used to process streaming data. The longest common increasing subsequence of two given input sequences is simply calculated. Based on the recursive equation, a simpler algorithm is found, and the new state is used to connect the incremental data. This method, which is applied in the manufacturing industry recommendation system, can respond to changing behaviours of users and adjust the ranking of recommendation results in real time, which can continuously improve users’ experience in the recommendation system.