A novel approach to label road defects in video data: semi-automated video analysis

There exists an extensive amount of studies, which present methods and systems to detect road hazards automatically. One of the most cited and famous systems is ‘Pothole Patrol’, published by MIT researchers in 2008 (Eriksson et al., 2008). The authors utilized an inertial sensor placed in the vehicle to detect potholes with a filter-based data processing pipeline. They managed to cover 2,492 distinct kilometers during their 10 days driving period. Overall, the detector misidentified road features as having potholes in less than 0.2% of the time, while 90% of reported detections contain actual road anomalies in need to repair. Most of studies employing inertial sensors have mainly tried to detect potholes or the overall road roughness. There are several approaches in recent literature to use camera systems to detect further types of road defects besides potholes, such as cracks or patches (Radopoulou and Brilakis, 2016). Hereby, supervised learning is the most prominent method to process the images from camera systems since it needs less training data compared to unsupervised learning. For example, Wu et al. (2014), Xu et al. (2008), and Zhou et al. (2006) applied artificial neural network algorithms to detect cracks on the road, which are trained with annotated images of road hazards. Radopoulou and Brilakis (2016) used semantic texton forests (STF), supervised learning algorithms, to segment road hazards based on their texture, layout, and context. Support vector machines (SVM) represent another supervised classification method for road condition monitoring, applied by Lin and Liu (2010).

Maeda et al. (2018) were the first to introduce a comprehensive labeled image data set for road damage recognition. The damage types are separated into the four main classes longitudinal cracks, lateral cracks, alligator cracks, and severe damages like potholes or bumps. Additionally, the classes ‘crosswalk blur’ and ‘white line blur’ are introduced to prevent confusion between these visual effects and actual road damages. In overall, 163,664 captured images they labeled 15,435 events but only 409 (2.65%) of these events were severe damages (e.g. potholes). This low number is caused by the low likelihood of occurrence of potholes in well-developed countries. Their proposed machine vision algorithms reached an accuracy of 79% for longitudinal cracks, 93% for lateral cracks, 84% for alligator cracks, and 95% for potholes.

Angulo et al. (2019) added 1,200 potholes and other road defects from Mexico and Italy to the Japanese data set in order to reach a more balanced number of events per class. They gathered 180,345 road images containing 45,435 instances of surface damages. Their accuracy of pothole detection increased to 98% using the new image data.

Results of recent studies show that cameras in combination with supervised learning methods can precisely detect and mark defects within images. However, one problem arises throughout the literature of camera systems and road condition monitoring. Composing a training data set is typically a highly manual, time consuming, and costly process. Researchers must go through a large size of acquired video material, select images with road defects, and annotate these events. In addition to this, manually labeled data sets will only include a small proportion of severe defects like potholes, sunken manholes, and bumps due to their low likelihood of occurrence. Creating a data set for combining acceleration data and video data is even harder because the operator must verify that the road defect was hit by the tyres.

To close this research gap, we propose a method to select images with road defects from acquired video material automatically. Our approach is based on an IMU sensor in synchronised operation with a camera. First, we show a method to fuse inertial sensor and video data in order to filter them automatically and detect road defects based on the inertial sensor data. Afterwards an algorithm selects the corresponding images with the identified road defects. In the last step, the user verifies and labels the road defects in the suggested images. This semi-automated process constitutes a substantial time saving over existing methods.

To gather a massive amount of unlabeled data, the University of Berkeley fitted a fleet of cars with smartphones mounted to the front windows. The data were gathered in urban areas of the east coast of the USA. It contains all types of weather and both day and night drives. The smartphones gathered acceleration, gyroscope, GPS, and video data of each drive. These data are separated into 20 to 30 sec segments to lower individual file size and to protect the privacy of the drivers.

We import the sensor data and pre-process it in order to minimize measurement inaccuracies.

First, we smoothen the acceleration data using a Savitzky–Golay filter

SG Savitzky–Golay filter taken from the Python scikit library.

Then we reduce the noise of the raw gyroscope data with a one-dimensional median filter. Typical measurement errors like spikes were thereby removed.

The program needs to operate with the information from all three sensors. To merge the data, we interpolate the gyroscope and location data to fit the timestamp vector of the accelerometer. This is done using one-dimensional linear interpolation

1D interpolation taken from the Python numpy library.

Ward and Iagnemma (2009) showed that the response of the vehicle to the excitation of the road depends on the driving speed. The conversion of a time series into space domain reduces this effect. To convert the data into space domain, we initially calculated the traveled distance for each time step using the following formula: Δ s = v ⋅ Δ t .

In the second step, we resample the data by distance using the same linear interpolation function as before. It would also be possible to calculate the space domain using the GPS coordinates, but this would lead to higher deviations due to the inaccuracy of the sensor.

The next process reduces the amount of data and extracts important information from the oscillation profiles. Our program compresses the data into so called windows. The data compression algorithm is described in Figure 2. The program combines five meters of data (500 data points) to one new data point. Each new point gets assigned the location and speed of the segment center point. Furthermore, the routine calculates the standard deviation of the acceleration in z-direction (std-acc-z) and the standard deviation of the angular velocity around the y-axis (std-gyro-y) for every window. To guarantee that every event is captured with this method, the windows should overlap.

Figure 2:

We want to estimate the road roughness by comparing the standard deviation of the z-acceleration with the minimum and maximum values of all data. The algorithm in Figure 3 describes how we calculate the scoring table, including the min/max values.

Figure 3:

The most important feature for the classification of road damages is the speed of the vehicle. The amount of vibration differs greatly with changing speeds.

To handle this dependency, we separate the data into sections of speed. The algorithm uses the entries from every drive to calculate the minimum and maximum values of the standard deviations. It is important that the entries are divided into their speed sections. Since there can be heavy outliers, the program sets the maximum entry to the 97% quantile of the standard deviation. The score table likewise includes the standard deviation of the gyroscope sensor data. The process is repeated for every ride and every section of speed.

The program is now able to calculate the specific score of a single window using the score table. We estimate the score by comparing the sensor data of a window with the minimum and maximum values of the corresponding speed section in the score table. The score is limited to lay in between 0 and 1. We determine the final score out of the mean of the acceleration and the gyroscope score. Figure 4 displays the required steps for the score estimation.

Figure 4:

The sensor data are in a json file format and contain the following features:

Acceleration in x, y and z-direction with a sampling rate of 50 Hz.

Each ride segment has one sensor data json file and a corresponding video file that matches its length. The video data have a sampling rate of 30 frames per second (fps).

The json file gets imported into Python and the data pre-processed as described in Chapter Data acquisition and signal processing. As parameters of the Savitzky–Golay filter, we choose a window length of 11 and a polynomial order of 5. The window length of the median filter is 5 m. The algorithm resamples the data into the space domain with a frequency of 100 m⁻¹.

The data get compressed to windows as shown in Chapter 2.2. To guarantee good coverage of the street, we set the windows length to 5 m and the overlapping factor to 66%. The compressed data are saved with the corresponding video file names for later use.

The score table, presented in Chapter Data compression, is build using the speed ranges shown in Figure 5.

Figure 5:

The created table helps to compare the sensor data with all previous data. Hence, we can assign a score to each window. Figure 6 displays the minimum and maximum values of the score table for an example data set. It is noticeable, that the z-acceleration value is high for medium speeds. The IMU vibration in the car increases with higher speeds. Typically, higher speed roads have fewer road defects and the vibration peaks are therefore lower. A similar pattern can be seen in the gyroscope data. The pitch y-gyroscope values at low speeds are higher than the corresponding vertical z-acceleration values. A possible explanation for this is the pitch movement a car makes when harsh cornering or braking. These movements directly lead to a higher y-angle velocity.

Figure 6:

After the data pre-processing, the video sequences can be analyzed. We use a threshold to distinguish the special events from normal driving. Every window with a score higher than a threshold T indicates a possible road defect.

To reduce the number of incorrect images presented to the operator, the program filters out every event with a speed lower than 3 m/s. If the vehicle velocity is lower than this threshold, the algorithm often detects harsh braking or acceleration as a road defect event.

Since the goal of the video analysis is to generate training data for an image classification, night drives should not be considered. This is simply done by measuring the mean grey level of the event image and filtering out all events with a value lower than 55 on a grey scale from 0 to 255.

The overlapping windows lead to multiple detections of the same event. We decided to only show the first event occasion, if the score value is higher than the threshold multiple times in a row.

The goal is to show the operator an image of the road defect. This cannot be done using the same image, as where the sensors measured an event, because the car is positioned over the road defect at this point. To get the correct camera image, the program must show a video frame, that lies some frames before the event occurred in the sensor data. Therefore, the routine calculates the number of frames by going back 9 meters in the video data using the fps of the video footage and the current speed of the vehicle: n f r a m e s = 9 m ⋅ fps v c u r r .

To better detect the damage cause, the program shows a short video sequence of the drive over the obstacle. After this, the user decides if the image contains a noticeable road defect and saves the event image.

To label the events in the images, we use the external program ‘labelImg’ (Lin, 2020). In Figure 7 you can see the image annotation process with ‘labelImg’. In this example, we identified two manholes and one sunken patch.

Figure 7:

The results in the previous chapter show that the labeling process for image recognition in road defect prediction can be sped up by a substantial margin. Applying the presented methods results in a more balanced training data set. Angulo showed that this can increase the accuracy of machine learning algorithms.

An event that was found with the presented approach is ensured to have correlated visual and acceleration data. Both Maeda and Angulo chose to only save one image per second and search those for damage events. This approach is not suitable for training advanced continuous algorithms like object tracking, visual severity measurement, or sensor fusion applications. Our method is optimal for generating ground truth data for these tasks.

The semi-automatic labeling is advantageous to the manual approach for several reasons. The presented method is much quicker because you skip all video sequences where no road defect is measurable. We were able to go through big data sets and search for high impact events like potholes, speedbumps, and railroad tracks very efficiently.

On the one hand, our approach results in a lot of severe defects detected which is very important for road event image classification. On the other hand, the manual operators were able to detect smaller and low impact events like cracks and patches more frequently. Those events tend to occur very often but have a lower score than events like potholes and fall under the threshold. It can be advantageous to label a few street kilometers manually in order to generate a high number of low impact events.

The operator has the choice to set the detection threshold T according to their data. It is recommended to pick a low threshold if you have limited amount of high-quality data in order to cover as many events as possible. If the data set is big and not professionally recorded, it is better to pick a slightly higher threshold.

The operators reported that the semi-automatic approach was less tiring and more rewarding. The use of a computer with a touchscreen and smart pen is recommended in both cases due to the more natural workflows.

Based on the described method the road inspection of a 5,000 km road network can be monitored by having several IMU and cameras traveling on many vehicles covering the road network regularly. With the proposed optimized, semi-automated analysis process road defects can be quickly mapped and assessed. Once they are prioritized, a road maintenance crew can be automatically dispatched to fix the identified road defects. This process can save up to an order of magnitude in data collection and analysis and cover hundreds of kilometers per unit vs. the current lower distance as described above.

With the proposed method we will be getting closer to preventative road maintenance and can ensure safer and smoother travels. Given the current inefficient spent on road asset monitoring and reactive maintenance, this will be a substantial commercial opportunity.

N3T has started to incorporate the findings of this paper in its improved road defect analysis process. We have built a Smartphone App which uses an z-acceleration threshold to capture an image after winding back the frames of the camera to where the road defect can be seen. The image is automatically sent to a cloud-based web database for near real-time analysis. Further automation of the current human labeling process will be investigated using transfer learning and other deep learning methods.