MarQR technology for measuring relative displacements of building structure elements with regard to joints and cracks

The pictures were taken using a digital camera Canon EOS R with a lens Canon 100 mm 1:2.8 L IS USM. The focusing distance was determined and unchanged throughout the calibration process as well as measurement exposure. The value of aperture was set for 32 to obtain the maximum depth of field possible and thus ensure the flexibility in choosing the camera distance to markers. The sensitivity of the camera matrix was set for the value of ISO 1000, which was to ensure good exposure conditions with an acceptable noise level at image recording. To ensure uniform lighting conditions from different camera stations, all pictures were taken with a flashgun Stroboss 60c.

Camera calibration in our case was performed with the help of a pattern consisting of regularly placed dots 18 mm in diameter and spaced out 25 mm from each other (Fig. 3). The pattern was printed on an A1 size foil and stuck on a glass sheet. To calculate calibration parameters, 20 pictures of the pattern taken from different camera stations were used.

Figure 3

Calibration points were the ‘centres’ of visible dots. For the sake of process accuracy, perspective distortion of the aforesaid ‘dots’ image, usually present in such cases, was taken into account.

For the sake of calibration of images (pictures) geometry, a model of lens distortion was used, which takes into account its radial and tangential distortions. For each pixel (u, v) of an image with a removed distortion, the function calculates the coordinates in the original image (x, y). The process is used as follows: x=u−cx′fx′,y=v−cy′fy′, \matrix{{x = {{u - c_x^{'}} \over {f_x^{'}}},} \cr {y = {{v - c_y^{'}} \over {f_y^{'}}},} \cr} where cx′ c_x^{'} and cy′ c_y^{'} are the target projection of the lens centre point on an image with removed distortion; fx′ f_x^{'} and fy′ f_y^{'} are the target image distance for an image with a removed distortion.

Radial and tangential distortion is removed according to equations: r2=x2+y2,x′=x1+k12+k24+k361+k42+k54+k66+2p1xy+p2(r2+2x′2),y′=y1+k12+k24+k361+k42+k54+k66+2p1(r2+2y′2)+p2xy, \matrix{{{r^2} = {x^2} + {y^2},} \cr {{x^{'}} = x{{1 + k_1^2 + k_2^4 + k_3^6} \over {1 + k_4^2 + k_5^4 + k_6^6}} + 2{p_1}xy + {p_2}({r^2} + 2{x^{'2}}),} \cr {{y^{'}} = y{{1 + k_1^2 + k_2^4 + k_3^6} \over {1 + k_4^2 + k_5^4 + k_6^6}} + 2{p_1}({r^2} + 2{y^{'2}}) + {p_2}xy,} \cr} where [k₁, k₂, k₃, k₄, k₅, k₆], and [p₁, p₂] are radial and tangential distortion parameters respectively. Finally, we map intensity levels onto the target image mapx(u,v)=x′fx+cx′,mapy(u,v)=y′fy+cy′. \matrix{{{\rm map}_{x}(u,v) = {x^{'}}{f_x} + c_x^{'},} \cr {{\rm map}_{y}(u,v) = {y^{'}}{f_y} + c_y^{'}.} \cr} where f_x and f_y are focal length in x and y direction of the original image, cx′ c_x^{'} and cy′ c_y^{'} are the target projection of the lens centre point. Additionally, we assumed that the original and target focal length is constant: fx′=fx,fy′=fy, \matrix{{f_x^{'} = {f_x},} \cr {f_y^{'} = {f_y},} \cr} and the lens center point (c_x, c_y) is in the middle of the target image: cx′=12(image_width−1),cy′=12(image_height−1). \matrix{{c_x^{'} = {1 \over 2}({\rm{image}}\_{\rm{width}} - 1),} \cr {c_y^{'} = {1 \over 2}({\rm{image}}\_{\rm{height}} - 1).} \cr}

Calibration calculations for individual sets of calibration pictures were made with the use of OpenCV procedure library (OpenCV, 2018). Corrected pictures were compared with their original images and residual error was determined. The average re-projection error of points on the original image did not exceed 0.5 of the pixel size (Fig. 4). Such calibration procedures allow to correct recorded images and continue their further processing in accordance with the photogrammetric procedures.

Figure 4

QR codes were adopted as markers to be used in the MarQR system, because of their suitability for multipoint raster analysis. Their key advantages are distinctive high-contrast geometrical features that are crucial for precise markers detection, combined with the ability to carry arbitrary information (Figure 5). This allowed markers to be self-identifiable by containing its name (number) and size encoded in its shape. This makes them perfect for automated and reader-independent monitoring systems. QR codes were generated and decoded with the use of ZBar library (Brown, 2011).

Figure 5

For the sake of tests concerning the results repeatability and system operational accuracy, the markers were printed on adhesive paper and stuck on a glass sheet to ensure their flatness.

As a result of the photogrammetric analysis of a picture set of QR code markers, their spatial interrelationships are set down. It is a two-stage process illustrated by the diagram in Figure 2. The first stage is processing individual pictures. It consists of appointing internal orientation elements of the camera and correction of lens distortion, finding the QR markers photographed on a picture and setting down the precise 2D position of markers’ apexes on an image. The second stage is determining the relation between markers based on two or more pictures. It consists of verifying the compliance of markers present in the pictures, setting down external orientation elements and optimising the mutual position of markers. The assumption was that the marker with lowest number was recognised as a reference one, and all markers’ relations are expressed in its coordinate system.

By applying the calculation algorithm, we obtain the values of translation and rotation parameters (6 degrees of freedom) of the measurement marker in relation to the referential one.

In the picture with a removed distortion, there are set down positions of QR markers. The procedure library ZBar automatically discovers markers on a given picture, decodes their textual content and shows their approximate position. Knowing the information coded in the marker, we can build its theoretical model. Owing to error correction built in the QR Code technology, the proposed algorithm of solving the problem is resistant to minor dirt or damage of the marker’s image.

In order to set down the precise position of the marker on a picture, an iterative algorithm is followed, concerning the adjustment of QR code’s image to a theoretical one by applying perspective transformation. The quality of adjustment is defined based on the normalised information shared by both NMI (Normalized Mutual Information) images (Figure 6) (Strehl and Ghosh, 2003): NMI(X,Y)=I(X,Y)H(X)H(Y), {\rm{NMI}}(X,Y) = {{I(X,Y)} \over {\sqrt {H(X)H(Y)}}}, where I(X, Y) is the mutual information concerning random variables X and Y defined as: I(X,Y)=∑y∈Y∑x∈Xp(x,y)logp(x,y)p(x)p(y), I(X,Y) = \sum\limits_{y \in Y} {\sum\limits_{x \in X} {p(x,y){\rm{log}}{{p(x,y)} \over {p(x)p(y)}},}} where p(x, y) is the mutual probability distribution of X and Y and the values p(x) and p(y) are, respectively, the occurrence probabilities in X and Y. Variables distribution H(X) and H(Y) are the entropy values of variables X and Y, respectively: H(x)=−∑i=1np(xi)log2p(xi). H(x) = - \sum\limits_{i = 1}^n {p({x_i}){\rm{lo}}{{\rm{g}}_2}p({x_i}).}

Figure 6

The coordinates of the marker’s four apexes were optimised parameters used in the present study. Applied optimisation was based on a nonlinear method of conjugate gradients. It consists of solving a nonlinear unconstrained problem (Hestenes and Stiefel, 1952) in which the vector value of optimised attributes x_k in subsequent iterations is determined from the dependence: xk+1=xk+αkdk, {x_{k + 1}} = {x_k} + {\alpha _k}{d_k}, where the step value α_k is determined by seeking an optimal position in direction d_k. Direction d_k is determined based on the dependence dk+1=−gk+1+βkdk, d0=−g0, \matrix{{{d_{k + 1}} = - {g_{k + 1}} + {\beta _k}{d_k},} \hfill \cr {\,\,\,\,\,\,{d_0} = - {g_0},} \hfill \cr} where β_k is the updating parameter of the conjugate gradient method and g_k = ∇f(x_k)^T is the gradient ∇f(x_k) of function f in x_k, where x_k is the row vector and g_k is the column vector.

Various conjugate gradient methods have different values of scalar β_k (Hager and Zhang, 2006). The value of parameter β_k was determined in accordance with Hestens and Siefel’s original study (Hestenes and Stiefel, 1952): βkHS=gk+1TykdkTyk, \beta _k^{HS} = {{g_{k + 1}^T{y_k}} \over {d_k^T{y_k}}}, where yk=gk+1−gk. {y_k} = {g_{k + 1}} - {g_k}.

To ensure optimisation stability, the process was carried out for many resolution levels of marker reconstruction, starting from the picture of 108 pixels × 108 pixels up to 540 pixels × 540 pixels. In this way, we managed to avoid local minimums, which potentially falsify the result of image adjustment. By applying the algorithm of the theoretical adjustment of the marker’s image to its photo, we obtain precise position of the markers’ vertices. On the basis of their positions and camera parameters determined during the calibration process, we can set down the position of the camera concerning the marker. In other words, it denotes determining the elements of external camera orientation. However, it should be noted that, in many cases, geometric conditions and a small marker size may lead to significant errors in determining the position of markers in 3D space. Applying several photos taken from different locations allows removing this defect effectively.

Owing to the application of many photos, it is possible to improve the accuracy significantly in determining mutual markers positions. The proposed concept is not limited by the number of photos based on which transformation parameters are determined. The smallest number of photos suggested by the authors is two (stereogram).

In the presented analysis, we concentrated on the case of two convergent photos. Such pictures allow designating both the position of projection centres of individual pictures and coordinates of the principal points of QR code images. The task of optimisation was carried out by the conjugate gradient method according to Hestenes and Siefel. The optimised vector of attributes included the position of the camera concerning the referential marker for each of the pictures and the relation between the referential and the measurement marker, which remains the same for each of the pictures. As a quality parameter of image adjustment, an average square error of apexes position determined in the picture in relation to markers’ projected position was adopted, which was carried out based on the adjustment: SQError=∑j=0n∑k=03(xjk2+yjk2), {\rm SQError} = \sqrt {\sum\limits_{j = 0}^n {\sum\limits_{k = 0}^3 {(x_{jk}^2 + y_{jk}^2)}}}, where k is the number of the marker’s vertices and j is the picture number.

It should be noted that the position of the referential marker in relation to the camera is influenced by the adjustment of all visible markers. It means that it is possible to increase the precision of determining its position by increasing the number of markers in the picture, which has been experimentally analysed by the authors of the present study.