Novel synaptic architecture for brain inspired computing
Consider the following. Our brain and all its magnificent capabilities is powered by less than 20 watts. Stop to think about that for a second. As I write this blog my laptop is using about 80 watts, yet at only a fourth of the power, our brain outperforms state-of-the-art supercomputers by several orders of magnitude when it comes to energy efficiency and volume. Nature is truly remarkable.
For this reason it shouldn’t be surprising that scientists around the world are seeking inspiration from the human brain as a promising avenue towards the development of next generation AI computing systems and while the IT industry has made significant progress in the past several years, particularly in using machine learning for computer vision and speech recognition, current technology is hitting a wall when it comes to deep neural networks matching the power efficiency of their biological counterpart, but this could be about to change.
As reported last week in Nature Communications, my colleagues and I at IBM Research and collaborators at EPFL and the New Jersey Institute of Technology have developed and experimentally tested an artificial synapse architecture using 1 million devices — a significant step towards realizing large-scale and energy efficient neuromorphic computing technology.
When the brilliant scientist John von Neumann built today’s computer architecture, which powers nearly 100% of the world’s computers, he kept the memory and the processing separately. This means data needs to constantly shuttle back and forth, generating heat and requiring a lot of energy – it’s an efficiency bottleneck. The brain of course doesn’t have different compartments, which is why it’s so efficient. But this didn’t deter teams from sticking with von Neumann’s design to build a neural network and while they have some success, the efficiency of these systems remains low – you simply can’t beat nature.
More recently, scientists, including those at IBM, have taken a different approach based on a new class of nanoscale devices known as memristive devices, which have shown great promise in addressing this bottleneck. Our device design is based on something called phase change memory (PCM), arguably the most advanced emerging non-volatile memory technology. An electric pulse is applied to the material, which changes the conductance of the device though its physical properties.
As explained in our paper:
Memristive devices such as PCM devices store information in their resistance/conductance states and exhibit conductivity modulation based on the programming history. The central idea in building cognitive hardware based on memristive devices is to store the synaptic weights as their conductance states and to perform the associated computational tasks in place. However, precise modulation of the device conductance over a wide dynamic range, necessary to maintain high network accuracy, is proving to be challenging.
Our breakthrough is in our design, which we call a multi-memristive synaptic architecture. This architecture enables us to increase the synaptic precision without increasing the power density even though we use several memristive devices to represent one synapse. The trick is that we have a nice selection mechanism, based on a global counter, which tells the device that it needs to change and when. The only penalty or cost is the requirement for more real-estate for the additional PCM devices.
To test our architecture, we trained both spiking and a non-spiking neural networks. Our data selected is a popular one – the MNIST data set of handwritten digits and our task is handwritten digit recognition – essentially our network needs to recognize what number is appearing from the handwritten images. In both cases, we see that the multi-memristive synapse significantly outperforms the conventional differential architectures with 2 devices, clearly illustrating the effectiveness of the proposed architecture. A highlight of the work is an experimental demonstration of the multi-memristive synaptic architecture in a spiking neural network using more than 1 million phase-change memory devices.
The architecture is applicable to a wide range of neural networks and memristive technologies and is crossbar-compatible. The proposed architecture and its experimental demonstration are a significant step towards the realization of highly efficient, large-scale neural networks based on memristive devices with typical, experimentally observed non-ideal characteristics. Having said that, we are also focused on improving the memristive device itself, it’s accuracy and dynamic range and then we think we can aim for the holy grail – floating point performance.
Neuromorphic computing with multi-memristive synapses, Irem Boybat, Manuel Le Gallo, S. R. Nandakumar, Timoleon Moraitis, Thomas Parnell, Tomas Tuma, Bipin Rajendran, Yusuf Leblebici, Abu Sebastian and Evangelos Eleftheriou, Nature Communications, volume 9, Article number: 2514 (2018)