1/10th scale autonomous vehicle based on convolutional neural network

The functional requirements of the system are analyzed and discussed in this section of the paper. Each component and its interfacing with the complete system is a crucial aspect of the vehicle.

Embedded computing platform: the primary requirement of the processor is to compute real-time data provided by the sensors. It needs to have sufficient RAM and CPU as performance speed is a crucial aspect of the system. It also needs easy interface ability with the different electronics and sensors used. The software compatibility requirements should also be able to meet the processor board since various machine learning functions will be run on it.

This section gives the complete circuit diagram implementation of the vehicle components. Firstly, the connections are described, and the interfacing pin configurations are explained thereafter.

Figure 2 shows the complete system circuit diagram developed for the vehicle. The electronics software tool Fritzing has been used to generate the circuit architecture.

Raspberry Pi and Camera: the camera is inserted in the Rpi camera slot through a 15-pin ribbon connector which attaches it firmly to the Rpi.

Figure 2:

The following section elaborates on the CAD designs used to build the vehicle base plate for housing the components. The first design is done for a base plate to house the raspberry pi, power bank, ultrasonic sensor, and PCA9695 servo driver. The base plate has been custom designed for this application. It is possible to use different materials and techniques to build the base plate like 3D printing or drilling. While these techniques will work, the fastest and most efficient one was to use a laser cutting technique Nag et al. (2018). In this system, an acrylic sheet of 4 mm thickness has been laser cut. The 2D CAD model has been made using Autodesk Fusion 360 design tool. Figure 3 shows the CAD model designed using this software. The design has been implemented such that it can be placed on the four corner sections of the chassis and also has sections laser cut to allow the other components to be fixed in place. Some of the components which need to be removed frequently like the power bank are fixed in a temporary manner as shown later.

Figure 3:

This design has further been implemented on the transparent acrylic sheet material with dimensions length = 360 mm, breadth = 190 mm, and thickness = 4 mm. The laser-cut process took only 15 min, while a similar 3D print this would take about 4 hr to complete.

The next CAD model has been designed to place the camera at a certain height and angle on the base plate body. The camera is to be positioned so that it can have a complete view of the lane markings. The tower-type case has been also made suitable to fix itself permanently with screws to the base plate and the camera as well. To make the case, a 3D CAD design has been made using Autodesk Fusion 360. Figure 4 shows the CAD design model along with its 3D printed model. An 18% infill with PLA material is used to make the physical case. The final dimensions are length = 150 mm, breadth = 85 mm, and height 85 mm.

Figure 4:

In this section of the paper, the software installation and configuration are discussed. The given circuit diagram is used, and the raspberry pi computer software is configured for the self-driving system. The microprocessor Rpi and a host Windows computer are used to perform these operations. Figure 5 shows an overview of the software message queue loop of the system. This overview gives the vehicle state in different stages.

Figure 5:

The first stage is where the vehicle is in ‘user mode’ or ‘training mode’. As can be seen, the first function is performed by the camera visual input where it captures images and saves them to the SD card. The control here is completely performed by the user through a joystick/web interface designed to communicate wirelessly with the onboard Rpi. In this mode, the vehicle is ‘trained’ by the user to drive on the track and take images that are saved on the memory card. The quality of images in the dataset is dependent on how well the driver can drive the vehicle on the path without any errors. The dataset contains images of the track with a timestamp and adjacent steering and throttle angles given for the motors by the controller while driving.

In the next stage, these data are trained in a CNN (convolutional neural network) model on the host PC. The generated machine learning model is then transferred to the raspberry pi. This stage is called the ‘autopilot mode or ‘test mode’ where the vehicle is tested to run without any human intervention or control on the same track. This time, the camera takes images, and the CNN model analyses these images in real-time and also outputs steering and throttle angle values. These values are sent from the Rpi to the motors in the form of PWM signals of varying duty cycle (on/off) period through the PCA9685 driver. The test data are also stored in the memory card and later can be analyzed with the training data. Here, the vehicle makes the best attempt to ‘replicate’ the user driving, hence the autopilot efficiency relies on the training datasets. This technique is also called supervised learning or behavioral cloning.

Host PC installation

In this section, the software installation for the training computer is provided as shown in Figure 6. Before the Rpi is configured, the software has to run through the PC. In this project, a Windows Laptop Nvidia GeForce GTX 1050 Ti GPU is used for this purpose. It is better to have a system with a powerful GPU to reduce training times. The software language used primarily for the configuration is Python. It is a widely used programming language for machine learning and AI applications.

Figure 6:

Firstly, the open source Python distribution platform ‘Anaconda’ is installed on the computer. The latest python 3.7 version is installed here.

To install the next software packages, the anaconda prompt terminal is used. Next, install the version control system ‘Git’ to manage the project and files on the computer. Next, the open-source code from various online sources is downloaded and managed in a separate folder on the laptop. Mainly, the GitHub repository from Tawn Kramer’s ‘Donkey Car’ (Roscoe, 2020) is used to build and configure the installation and software process. Next, the TensorFlow which is an open-source software library is installed with version 1.13.1. It also provides support to run ‘Keras’, the open-source deep-learning library to train the machine learning models. The majority of the software programming done in this project uses open-source material to configure the self-driving capability of the vehicle. However, the programs used have been configured and updated for this specific application and algorithm discussed later.

Raspberry Pi setup

The next step is the configuration of the Rpi on board the vehicle. Figure 7 shows the eight steps in the process of configuring the Rpi with the help of the laptop commands. These steps are to be performed only once to permanently configure the Rpi board.

Figure 7:

Initially, the micro SD card is connected to the laptop and the latest Raspbian Buster OS is flashed. Next, the Rpi needs to be enabled to perform operations from commands given by the laptop. For that, the Wi-Fi login and ID/password is saved in the memory card as a ‘.conf’ file and used by the Rpi to connect to Wi-Fi on the first boot. Next, the Secure Shell (SSH) is set up by adding a file named ‘ssh’ in the root directory of the memory card. Now, this memory card is inserted in the Rpi and is ready to connect to the laptop via Wi-Fi. The connection uses the Host PC connection via local SSH ping. Also, the update and upgrade functions are performed from the terminal. Next, the camera and I2C for the Servo driver PCA9685 are configured with the ‘sudo raspi-config’ command. In the configuration, the default password, the hostname is changed and the ‘I2C’ and ‘Camera’ are enabled in the ‘Interfacing Options’. Also, in ‘Advanced Options’ the ‘Expand Filesystem’ is used to use the complete SD card memory storage space. Next, the necessary dependencies and software libraries for python, i.e. version, dev, pip, virtual environment, camera, numpy, pandas, gpio, i2c-tools, avahi-utils joystick, etc. are installed using the ‘sudo’ command. Next, the virtual environment is activated. Also, the Open CV for a real-time computer vision application is installed. Lastly, the files are managed, and a separate folder called ‘projectsv2’ is created.

A new file myconfig.py has been created for the vehicle’s main loop. This code contains all the necessary program and each sub-component class is used in this python script to execute it in real-time onboard the Rpi.

Camera

The camera connection is shown earlier in the circuit diagram. After enabling the camera interfacing option in the raspi-config, the camera can start functioning. The camera setup is done in a new file created named camera.py where a new class Base Camera is defined. This file is the main program in augvconfig.py that will be executed by the system while in training mode. To exit and save the file: press ‘Ctrl + Fn + X’ and then ‘Y’ and then press ‘Enter’ while in the ‘nano ~/mycar/myconfig.py’. The first step is to define the camera parameters and dimensions. The selected dimensions for the camera are width = 160, height = 120, and depth = 3. The depth is 3 for RGB color (Red Blue Green) if 1 is selected it is for mono. The height and width are in pixels. Other executable conditions like horizontal/vertical flip, the frame rate is also defined here. An fps of 20 is selected for this project. Figure 8 shows a screenshot of the camera code configuration and a sample image of 160 × 120 pixels taken from the dataset.

Figure 8:

PCA9685 servo driver

The servo driver used is connected to the I2C port of the Rpi and is enabled as mentioned earlier. The commands ‘sudo apt-get install -y i2c-tools’ and ‘sudo i2cdetect -y 0’ are used here to complete the setup. This gives the default address of the 16 channel PCA9685 board as 0 × 40 (binary 1000000) shown in the screen capture in Figure 9.

Figure 9:

This address 0 × 40 is used in the vehicle augvconfig.py file. Further, the steering channel is selected at 1 and throttle as 0, referring to the circuit diagram.

Web interface and joystick control

For this project, both the web interface and joystick setup have been performed. The PS4 joystick uses the Bluetooth connection with the Rpi to communicate and send commands. A separate python script joystick.py is created to define the class and commands for the ps4 buttons. The ds4drv is a Sony DualShock 4 user space driver for Linux which is installed using ‘sudo pip install ds4drv’ on the Rpi. After completing the setup and assigning buttons and axis controls to the program, the joystick can be used to control the vehicle.

The web interface is written using HTML and python languages to control the vehicle. Figure 10 shows the screen capture of the web interface connected through the IP address localhost.

Figure 10:

Here, the final vehicle system is assembled with its chassis, motors and ESC, complete circuit diagram, the 3D printed, and laser cut models and positioning of the camera and ultrasonic sensor. The following provides a step by step process that has been implemented for the vehicle.

Firstly, the DC motor, Servo Motor, and ESC is attached to the vehicle 1/10th scale chassis. The chassis comes prebuilt with holes and sections for the motors and ESC to be attached. The DC motor is attached to the gearbox on the chassis. Also, the wheels had to be attached separately to the chassis to complete the base. These components were fixed permanently with a screw. However, the NiMH battery is placed on the chassis and fixed with R-clips, which makes it easy to remove and attach for charging. Next, the laser cut acrylic sheet is assembled with the Rpi and PCA9685 which has been attached to it via cable ties. The power bank, breadboard, and ultrasonic sensor have been hot glued to the laser cut base to avoid them from falling at high speeds. Next, the wiring has been done according to the circuit diagram shown earlier. The heatsink is also added to the Rpi here to avoid overheating. The tower case has been printed with indents to fasten screws to the base through the four corners on which it stands on. Lastly, the camera connects to the front section of the tower case with screws. The breadboard has been used here to allow further prototyping of the system with additional sensors if required in future upgrades. Finally, the acrylic plate is placed on the chassis and fixed using R-Clip as shown. Figure 11 shows the complete assembly of the vehicle hardware.

Figure 11:

Before the vehicle is tested to run in user/autopilot mode, the circuit has to be tested with the software configurations and necessary calibrations are made.

Firstly, the DC motor connected to the ESC is calibrated with the Rpi. The primary aim of this testing is to find a suitable speed and subsequent PWM value to control the throttle rate. Also, the forward, reverse, and stop functions have been calibrated with the servo driver and Rpi. As mentioned earlier, the channel 0 of the PWM board is the throttle channel, and channel 1 is for steering. Figure 12 shows the code snippet program of the motor calibration used in the myconfig.py file.

Figure 12:

The calibration is crucial to avoid the car having too much speed and hitting a wall or going off track. The value of the PWM control signals pulse range from 1,000 to 2,000 microseconds. The width of the pulse determines the speed and direction of the motor rotation. Hence, the value 1,500 points to the center for the motor, i.e. stop signal. To find these values given in Figure 12, different values are given via the Rpi starting from 370 and going +/−10. The reverse calibration is tricky and takes a while to figure out since it requires the ‘reverse’ value, then the ‘stop’ value, and then again ‘reverse’.

Similarly, the steering values are calibrated where the control signal sends one PWM pulse at 60 times/sec rate. The width of this pulse decides the amount of right/left the motor turns. The steering servo results in its extreme left at 415 value and extreme right at 295 PWM value. The servo can go +/−60° in either direction corresponding to +1/−1 as shown in Figure 13. The floating point values between +1/−1 are used to save the angular control values in the datasets shown in Figure 14.

Figure 13:

Figure 14:

The following section describes the custom track built to train and test the vehicle. Building a custom track is a challenging task and getting it to work requires multiple iterations. Firstly, a large enough space is required to build the track as the car has a turning radius defined by the servo motor angle. The selected track has been built indoors in an apartment type space. Bright color ribbons have been used to make the track lanes and center markings and they are stuck to the floor using tape. Green color has been used for the inner and outer lane and white color ribbons for the broken center line markings. Figure 15 shows the custom-built indoor track design for the car and its implementation.

Figure 15:

The track dimensions are 15.2 meters (4.6 + 4.6 + 3+ 3) for the outside marking and 11.6 meters (3.8 + 3.8 +    2 + 2) for the inner marking. Thus, this provides sufficient space for the 1/10th scale car to move with a smooth turning space. Further, designing the track indoors is useful to allow frequent testing and adjustments.

Car training and testing is split into three sections.

Train/user mode

The car is driven on this track by the user using a joystick/web interface for about 15 to 20 laps. These training data are saved on the memory card containing several such images and ‘. json’ text files as shown in Figure 16. This dataset is then transferred to the laptop via SSH protocol using ‘FileZilla’ software.

Figure 16:

Host PC training

Training the dataset on the Keras model requires larger training times (couple of hours) for larger datasets. This dataset is trained on the host laptop using a custom CNN model with the following parameters:

No.	Convolution filters	Strides	FC layers	Parameters	Loss
1	12 × 3 × 3, 18 × 3 × 3, 24 × 3 × 3, 36 × 3 × 3	3,2,2,1	900, 246, 32	1,100k	0.098543

A batch size of 128 is used with Keras linear type model. The model has a validation loss of 0.0.98543 which is suitable for the autopilot. The model is saved as a ‘.h5’ file which then has been transferred back to the Rpi for running it in real-time. Figure 17 shows the model loss of the training data. It can be seen at about 55 epochs; the training stops as the loss starts increasing.

Figure 17:

Test/autopilot mode

In this mode, the trained model is used by the Rpi to run on the track in a continuous loop as shown in Figure 15. The following flowchart algorithm shown in Figure 18 is used by the autopilot mode. For this application, the throttle speed is kept at a constant value and only steering values are predicted by the autopilot.

Figure 18:

The vehicle loop begins with initializing the camera and ultrasonic sensors and running the python scripts for the car defined in manage.py. Once the car is switched on, it first checks if there is any obstacle within 10 cm in front of the car. If no, the car starts moving and the camera inputs the images at 20fps. These images are then analyzed by the CNN model provided and output a predicted value of steering angle between −1 and +1. The value is then set by the actuator and the servo motor turns or remains constant. In the actuation, the vehicle state also checks for any user inputs given to increase or decrease the throttle. If the car detects any obstacle in 10 cm range, it immediately sets the actuation throttle value to 0 and waits until the obstacle is removed.

The training dataset contained about 30,960 records of images and ‘. json’ file records. This dataset is analyzed here as shown in the first graph using Jupyter Notebook and Python programming language. The graph plot (A) in Figure 19 shows that the most common frequency of angle was about ‘0’, i.e. go straight. The car is driven on the track in a clockwise direction due to which the frequency of the right turn is quite larger than the left turns. In fact, the only time left-turning angles are used is when the car needs to align itself with the straight lane marking of the track.

Figure 19:

The second graph plot (B) shows the autopilot dataset analyzed after the car is run for the same number of laps as the training mode. The graph plots are almost identical proving the autopilot is successful in behaving as the driver with minimal errors.