Deep learning-based image classification for autonomous vehicles and training by Unreal Engine

Nowadays, there are many reasons to use a game engine other than creating a video game. One of those reason is becoming quite popular: creating a digital twin meant to be used in applying machine learning models. This master thesis falls within such a project. Our project is to create digital twins of urban scenes. . This master thesis focusses on a specific aspect of the project: allowing vehicle in the digital twin to move in an autonomous way, and more precisely to make the decision of braking or not, solely based on driver’s frontal point of view, speed and steering angle. Taking inspiration from the literature, we develop the action plan of our research that consists of three steps. The first step is comparing the performance of four convolutional neural networks – VGG16, VGG19, ResNet50 and Xception - with solely the driver’s frontal point of view as input. The second step is comparing the same models improved by adding numerical inputs, i.e. speed and steering angle. Both family of inputs are assigned a model – a CNN for the visual input, and an MLP for the numerical inputs – and then the models are combined in another MLP classifier to get final predictions. The third and final step is to boost the prediction performance by aggregating the predictions of the four models of the second step with ensemble techniques. On average, the Balanced Classification Rate (BCR) and accuracy have increased at each development step and ensemble learning has yielded the best BCR with 72.6% and the best accuracy with 78.2%.