Introduction to Neural Networks

Inspired by the workings of neurons in the brain, artificial neural networks were defined in the 1940s. The perceptron was introduced in 1957, followed by the multilayer perceptron in 1986. They disappeared from the news for a while, before coming back to the fore in the 2000s with deep neural networks.

The artificial neuron is presented, with its various activation functions. The operation of the single-layer perceptron is presented, with examples of the implementation of logical OR and AND functions, as well as its limitations. The characteristics of multilayer perceptrons are detailed, including the principles of gradient backpropagation, which enables the weights of neurons in different layers to be updated.

Deep neural networks marked a milestone in the development and use of neural networks. Convolutional neural networks (CNNs) are described in detail; the convolution and pooling layers precede the fully connected layers that enable data sizes to be restricted by extracting simple spatial features (such as contours), which subsequent layers combine to form more complex, abstract concepts: assembling contours into patterns, patterns into parts of objects, parts of objects into objects, and so on. This enables thousands of different configurations to be used as network inputs. The characteristics of recurrent networks, enabling recognition of input sequences, are also presented.

The implementation of neural networks has both software and hardware aspects:

• programming. Software libraries such as TensorFlow, Caffee, Torch and PyTorch can be used on multi-core CPUs with or without GPUs. The various operations are defined in the form of a graph: program execution places the various operations on CPU, GPU or any other specific processor or gas pedal and defines the methods for their execution;

• hardware support. Specialized hardware operators (convolution, pooling) are available as IPs for systems-on-a-chip (SoCs); neural processors are available (ARM neural processors, Google TPUs). In addition, reduced data formats such as 16-bit floats are integrated into neural processors or GPUs to reduce memory footprint and power consumption, and accelerate calculations.

Neural networks are increasingly used in embedded applications. The constraints of embedded components mean that the networks used have to be reduced and optimized. Four techniques are possible:

• network simplification/optimization: reduce the number of layers to reduce the number of parameters;

• pruning: reducing the size of a network by removing some of its parameters;

...