Overview of current techniques in remote data auditing

In an unreliable storage system, when data failures happen, redundancy plays an important role in improving the reliability. In order to effectively deal with data failures, the effective way is to use a replication technique in which multiple copies of data are outsourced within the distributed storage systems (see Ying and Vlassov [21]). When data corrupt, the client can use an intact copy of the file with size |f| from any of the r unaffected servers. However, using the replication method to take up the storage space is r|f| (see Agrawal and Jalote [22]).

Although realizes the method is simple, there is no strong evidence to prove that the cloud actually stores multiple copies of the data files. In other words, in the replication-based storage systems, we don’t know whether our data in the server has multiple backup (see Chen et al. [23]). Such as, in peer-to-peer networks, servers perform a freeloading attack with the aim of using disproportionately more system’s resources without contributing a proportionate quantity of resources back to peers (refer to Osipkov et al. [24] and Ateniese et al. [19]). As a result, the data owners encounter a decline in the durability and availability of the file and the CSP has more storage space to sell to the users.

A simple method is to allow the user to apply for a single Provable data possession (PDP) method t times for t different servers to overcome the above problems. But, the servers can collude and pretend that t copies of files are stored while only a single copy is stored in reality as an identical copy of the file is stored on all of the t servers (see Chen et al. [23]). Besides, using this method computational cost is too high and the method is inapplicable for distributed storage systems particularly for larger values of t.

Curtmola et al. proposed a provably-secure scheme, called Multiple Replica Provable Data Possession (MRPDP), which were the first to address the collude attack in the replication-based schemes. In this scheme, the client encrypts the original file and masks the blocks of the encrypted file by using a random value (Ru) for each of the replica to generates t unique replicas (Rs) (see Barsoum and Hasan [25]). Then, the client creates a tag using a decrypted file for each block. Storage form as shown below.

Ti=(h(ν||i).gb[i])dmodn,$$\begin{array}{} \displaystyle {T_i} = {(h(v||i).{g^b}[i])^d}mod\;n, \end{array}$$(1)

(Ti is the tag for the i^t h data block (b[i]), n is the number of blocks, d and ( are the clients private key, and g is the public key)

The user selected the method of random sampling to ensure the data in the servers. By the preceding description, we may draw the method which is suitable for checking the integrity of outsourced data in distributed servers. However, the method only supports private verification and when we update the file it will generate a huge computation and communication overhead on the client and server (see Barsoum and Hasan [25]).

A scheme named Efficient Multi-Copy Provable Data Possession (EMC-PDP) was proposed by Barsoum and Hasan. This scheme is based on the technique of Boneh-Lynn-Shacham (BLS) homomorphic linear authenticators. The scheme includes two versions: Deterministic (DEMC-PDP) and Probabilistic (PEMC-PDP). The deterministic version verified the file blocks and the probabilistic version casually inspects the file.

The main method of the scheme is to generate a unique replication of file by attaching a replica number to the original file. The client using the following equation generates a tag for each block of replicas and allocation the tag among different servers.

Ti,j=(h(Fid)ub[i][j])d,$$\begin{array}{} \displaystyle {T_i}{,_j} = {(h({F_{id}}){u^{b[i][j]}})^d}, \end{array}$$(2)

(F_id indicates a unique fingerprint for each file that is generated by attaching the filename to the number of blocks and the number of replica, i indicates the replication number, j is the block number, d is the client’s private key, and u is a generator for a bilinear group mapping G).Although this scheme support the trusted third party auditing test, to update a block of the file, the client must re-encrypt and upload the entire replicas to the servers. Even though this scheme offering the security, its produce a higher storage overhead not only on the client but on the server.

The Dynamic Multi-Replica Provable Data Possession (DMR-PDP) scheme was proposed by Mukundan et al. [26] verified the integrity of multiple copies. The scheme utilized the technique of the Paillier encryption to generate distinct copies of the original file F = b[1],b[2],...,b[m]. The process shown in figure 4 uses the following equation:

Fig. 4

Ri={(1+N)b[j](kiri,j)},$$\begin{array}{} \displaystyle {R_i} = \left\{ {{{(1 + N)}^{b[j]}}({k_i}{r_i}{,_j})} \right\}, \end{array}$$(3)

(j is the index of the block, i is the number of replicas, kjrepresents a random number that is used to identify the number of replica, r_i,_j indicates a random number that is used for the Paillier encryption scheme, and N is the owner’s public key )

Xiao et al. [27] using the homomorphic verification response (HVR) and hash index hierarchy (HIH), called the Cooperative Provable Data Possession (CPDP), built a replication-based remote data auditing framework for distributed systems.The HIH is a hierarchical structure includes three layers: Express, Service, and Storage Layers. They are described as follows:

Express Layer: offers an abstract representation of the stored resources;

Homomorphic verifiable response is used to integrate multiple responses from the different CSPs in CPCP scheme. Homomorphic verifiable response is the key technique of CPDP because it not only reduces the communication bandwidth, but also conceals the location of outsourced data in the distributed cloud storage environment. However, a heterogeneous structure of the proposed scheme leads to a high communication load due to the inter-communication between various cloud servers.

The erasure code technique is based on the Maximum Distance Separable (MDS) code which is a form of data repairing technique. The erasure code technique compared to replication, provides more reliability for the same redundancy.As is shown in the Figure 5, when a block of file corruption is detected, the client can utilize the remaining intact blocks to recompute the codes of the corrupted block. Figure 5 indicates the original input file including three blocks (b[1],b[2],b[3]) is encrypted using the (3:2) erasure code. However, the communication overhead of this method is higher than the replication-based technique (see Kubiatowicz et al. [28], Changho and Ramchandran [29], and Li and Shu [30]).

Fig. 5

The scheme of High-Availability and Integrity Layer (HAIL) is about erasure coding-based remote data auditing for distributed storage systems. When a file corruption is detected in a server, the client recovers the corrupted block using the Proofs of Retrievability scheme. This project consists of three parts as follows: (a) Dispersal code: using erasure code technique distributed the file to the servers. (b) Server code: when the servers receive the data blocks, the blocks need to be encoded using an error-correcting code to guarantee the blocks. (c) Aggregation code: the responses from all of the servers to the received challenge including the multiple Message Authentication Codes are combined into a single composite Message Authentication Code. Figure 6 shows the dispersal coding technique.

Fig. 6

The scheme of Secure Distributed Storage (SDS) is based on Homomorphic Token and the Reed-Solomon erasure-correcting code to guarantee the data integrity and availability in an erasure code distributed storage system (see Wang [31]). Compared to previous methods, the most distinctive feature of this method is to support error localization techniques. This project mainly consists of four parts, as follows:

Data Dispersal: Using the Reed-Solomon erasure code to distribute the original file to the servers and encrypt the blocks.

(4) Dynamic Update: The scheme supports the client dynamic update operations, such as updates, deletes, appends, and inserts. The client utilize homomorphic token updataes the changed blocks without retrieving other blocks.

The technique of the network coding has the capability to reduce the communication overhead during the repair process. When a block failure of file was detected, a new data block is created on the basis of a linear combination of the stored data blocks across the intact servers during the repair process. For example, the client is given a file including m bloks(F = b[1],b[2],...,b[m]), based on a coding coefficient vector of m random value (ν₁,ν₂,...,ν_m):

NC=∑i=1mνib[i],$$\begin{array}{} \displaystyle NC = \mathop \sum \limits_m^{i = 1} {\nu_i}b[i], \end{array}$$(5)

the original file as a linear combination of the data blocks is generated.As is shown in Figure 7: a network coding-based distributed storage system for the original file includes three blocks (b[1],b[2],b[3]). The network coding-based RDA methods must meet the following four conditions Juels et al. [12], Zhang et al. [34] and Oliveira et al. [35]:(i)Error localization (ii) Loss of a fixed layout of the file (iii) Reply attack (iv)Pollution attack.

Fig. 7

In Anh and Markopoulou [36], the authors built on a RDA method which relies on a homomorphic Message Authentication Code scheme and a customized encryption scheme. when the client and server have a shared secret key, the scheme existence risk of pollution attacks. The homomorphic Message Authentication Code cryptosystem protect the scheme avoid the risk and this cryptosystem consists of three parts, namely: sign, combine, and verify. We use the sign algorithm to generate a tag for blocks and then use the homomorphic property to generate a linear combination of tags of the blocks. The characteristics of the scheme to some extent reduced the communication and computation overhead. In order to reduce the burden of the user on communication and computation overhead, the verification tag of the new data block is computed by combining the verification tags of intact blocks. So, we don’t need to have the client generate a verification tag for the new data block. Figure 8 shows the scheme including setup, challenge, proof, and verify steps.

Fig. 8

In the static data auditing methods, the clients must download the whole outsourced data from the cloud and upload them after performing the corresponding operations, during the update operations, such as modify, delete, insert, and open. Supporting dynamic data update is an important characteristic of RDA methods both for single and distributed cloud servers, since many common applications such as online word processing intrinsically deal with dynamic form of data or involved with dynamic log files. If the auditing approach supports the only needs to download the number of blocks that are to be updated. Therefore, such a feature reduces the cost of computation and communication of updating data on the client and servers.

Although Cash et al. [37] put forward a method to overcome the dynamic data update issue in cloud computing, the lack of public verification feature makes this method impractical . On the other hand, current dynamic data update methods also impose high storage and computation overhead on data owner. As a result, enabling user to efficiently and dynamically update their outsourced data requires more future research and developments.

Because of the characteristic of the current RDA method, addressing batch auditing in the distributed storage systems is more difficult. For the TPA process multiple auditing tasks received from different users at the same time rather than performing each of the tasks one by one. In other word, batch auditing mechanism utilizes the linear sum of the random blocks to shorten the proof message and thus mitigate the corresponding communication overhead. Only a few current RDA methods focus on batch auditing issue in the distributed storage systems (refer to Agrawal and Boneh [38]). A simple way to achieve such a goal is to use a bilinear aggregate signature to combine the proof messages into a single and unique signature that can be verified by the auditor.

When data owner outsource data to the remote cloud or delegate the auditing task to the third party, under the hypothesis, the TPA is a trustworthy agent. Obviously, this is not a logical assumption increasing the possibility of leaks. Data owner does not want the TPA, of course, knowing the details of data contents (see Mandagere et al. [39] and Meyer and Bolosky [40]). How to effectively solve the problem of the development of the technology of the RDA is very important.

In the era of big data where billions of files of data are stored in the cloud. 2013 global data volume 4.4 ZB, global data in 2014 at around 6.2 ZB, total 2015 global total data at around 8.6 ZB, 2016 will be around 12 ZB, in 2020, the amount of data will reach 40 ZB. As data volumes growing exponentially, the communication, storage and computation cost on both the auditor side and the provider side to existing RDA technology faces great challenges. Nowadays, more and more big data applications employ cloud to store a mass of data, such as Facebook. Although every day the users of Facebook just to produce a small amount of text and image data, the user behavior records are constantly updated and the user behavior is very important in the era of big data (see Sakr et al. [41] and Naone [42]). The dynamic data that a large number of data owner modifies a single bit of the outsourced data brought a problem that data auditing worsens.