Embedded real-time inference in spiking neural networks for neuromorphic IoT vision sensors

In recent years, artificial intelligence has attracted a huge interest in image and speech recognition, promising the advent of intelligent technologies in everyday life. Machine learning and computer vision have received a growing interest for solving complex image recognition tasks but often require powerful platforms to operate [1, 2, 3, 4]. One of the next biggest challenges these techniques have to answer is to be integrated inside mobile devices such as smartphones, wearables and other Internet of Things (IoT) nodes [5, 6]. It enables the possibility of local processing of data for real-time applications and avoids relying on cloud computing servers. In order to achieve this goal, it is believed that a paradigm shift towards alternative algorithmic and hardware solutions is necessary in order to lower the energy costs [7]. Inspiration from biological phenomena led to new approaches for treating information called neuromorphic processing. Neuromorphic event-based cameras [8, 9, 10, 11] mimic the process by which biological retinas send visual information to the brain, with asynchronous events when a change in brightness is detected by a pixel. New machine learning models such as Spiking Neural Networks (SNN) [12, 13, 14] have been developed by taking inspiration from the brain cell in order to process event-based information and achieve low power, low-latency and brain-inspired learning techniques [7, 15, 16]. The focus in this master thesis is to obtain a real-time embedded low-power implementation of a spiking neural network in an event-based computer vision application. The task is to classify samples of the challenging N-CARS [17] dataset on an embedded low-power platform and achieve minimal latency. The embedded application is developed in software and targeted to the ARM Cortex-A9 processor on a Cyclone V SoC. An accuracy of 88.59 % was obtained on the N-CARS dataset with a convolutional SNN based on the Slayer model [13], competing with state-of-the-art event-based low-power methods (HATS: 90.2 %) [17] and cloud computing approaches (EST: 92.5 %) [18]. A real-time implementation for the optimized model was achieved on the ARM Cortex-A9 processor for 16-bit and 32-bit floating-point data-types, with event-based convolution and dense operations. It demonstrated a 2 ms latency using 4-lane parallel NEON SIMD instructions and optimizes memory accesses with cache pre-fetching. The design is in competition with the HATS [17] FPGA implementation of Sethi et al. [19] with a 3.3 ms latency. This work additionally provides estimations of the execution time of the program in terms of the model parameters and number of events.