Computing in/near memory

For several decades now, the gap between processor and DRAM memory performance, known as the "memory wall", has been growing steadily. Various techniques are used to limit the growth of this gap:

1. cache hierarchies, to bring instructions and processor data closer together;

2. hardware multithreading to limit memory requirements ;

3. increasing DRAM throughput with successive generations: DDR, GDDR, HBM.

Bringing calculations closer to memory data is a technique that has been studied since the 1960s. Implementations such as Vector IRAM were proposed in the 1990s. In-memory or near-memory computing is becoming more topical as a result of two phenomena:

1. Many modern applications use huge data sets. Minimizing transfers between CPU and DRAM main memory is becoming a must.

2. Hardware circuit design techniques, such as the 3D stacking of chips in DRAM HBM (High Bandwidth Memory), facilitate computing close to DRAM memories.

Calculating near or in memory raises a number of questions:

1. Where to calculate?

2. How much calculation is required?

3. How do you organize coordination between the master CPU and the hardware gas pedal in or near memory?

These questions are detailed.

Five recent examples are discussed:

1. The Untether AI Boqueria architecture is an accelerator for neural network inference. It consists of a 2D grid of 729 SRAM blocks, each block comprising 512 SRAMs of 640 bytes and 512 elementary processors. Calculations are close to SRAM.

2. The Celebras WS2 circuit is a deep learning wafer with 850,000 cores (2.6 . 10 ¹² transistors). The cores, interconnected in a 2D grid at wafer level, have a 50:50 ratio of logic (computation) and SRAM memory.

3. The Ambit project modifies the internal structure of a DRAM to perform a number of basic operations: copy, Not, And, Or, etc.

4. UPMEM has designed and tested PIM chips comprising a processor based on DRAM technology with a complete instruction set for full computation, without floats or SIMD instructions, alongside DRAM memory banks. Calculations are performed close to the DRAM memory banks.

5. Samsung's Aquabolt-XL circuit stacks DRAM chips using TSV technology and inserts chips with computing units between the memory banks into the stack. The calculation...