Hardware supports for Deep Neural Networks

With the growing importance of artificial intelligence applications, deep neural networks are being used more and more. They have seen the development of significant hardware and software support. Major operators (Google, Microsoft, etc.) and circuit suppliers (ARM, Intel, NVidia, Xilinx), as well as a host of smaller companies and startups, are offering hardware solutions to accelerate the execution of applications using deep neural networks. The aim of this article is to explain the characteristics of these hardware solutions in relation to the main features of neural networks.

Without claiming to be a theoretical or exhaustive presentation, the basic principles of neural networks (NN) are recalled: structure of a NN, structure of a neuron, activation function as well as the two phases of use of a NN (Learning and Inference). Neural networks are used at several levels: data centers, servers at the edge of the network (edge devices), smartphones and components of the Internet of Things (IoT) with different performance and energy consumption constraints, leading to different hardware supports.

While 32-bit floats are the basic digital format for neural networks, performance and power constraints have led to the use of 8-bit and 16-bit integer formats and reduced float formats (F16, BF16, TF32), which are presented. The specific operators of convolutional neural networks are presented: convolution, pooling, fully connected (dense) layers.

Examples of hardware support are presented: AI instructions from the Intel instruction set for integer computing, NVidia tensor cores, neural processors from ARM (Ethos), Intel (Nirvana NPP-T), Google (TPU) and the Xilinx VC 1902 system-on-a-chip.