Energy Efficient Mott Activation Neuron for Full Hardware Implementation of Neural Networks

Thermal driven gradual resistance switching of vanadium oxide can be used to replace complex circuits for activation function implementation in neuromorphic system with an energy efficient nanoscale device.

Like Comment
Read the paper

  In 2016, AlphaGo [1], Google DeepMind’s machine learning program, defeated 18-times world champion of the game of Go. This big achievement has increased the enthusiasm and led to further advances in the field of machine learning. Recently, a new program, MuZero, has demonstrated that machine learning algorithms based on neural networks can master various games without knowing the rules for the games [2]. Even though these algorithms based on neural network models have advanced to learn from ground zero, there is one characteristic that has never changed: they require data-intensive computing.

  The data-intensive computing characteristic of machine learning made conventional Von Neumann architecture obsolete due to its excessive delay and energy consumption arise from data transfer between memory and processor, so-called Von Neumann bottleneck. To circumvent this limitation, in-memory computing architectures have been proposed and demonstrated promising results [3][4]. Although in-memory computing architectures provide a solution toward minimization of the data transfer between memory and processor, non-linear functions (e.g. rectified linear function (ReLU), sigmoid, and tanh), which play an essential role in learning algorithms, are still implemented with large-scale and energy-consuming complex circuits or even using external general purpose processors [5]. In this work, we demonstrate a nanoscale device that can replace the complex circuits for non-linear functions while providing substantial gains in area and energy efficiency.

Fig. 1 a Printed Circuit Board (PCB) designed for integrating an eNVM device array with Mott ReLU devices. b A schematic shows how an eNVM device array is connected with Mott ReLU devices. c Scanning Electron Microscopy (SEM) image of fabricated Ag conductive bridge random access memory (CBRAM) array. d Schematic of Mott ReLU device. e SEM image of fabricated Mott ReLU device.

  Neuroelectronics group (PI: Prof. Duygu Kuzum) and Schuller Nanoscience group (PI: Prof. Ivan K. Schuller) at University of California San Diego came up with the idea of harnessing electronic switching characteristics of Vanadium oxide (VO2) for implementing one of the widely used non-linear activation functions, ReLU, in a nanoscale device form. VO2 exhibits an interesting electrical switching phenomenon, insulator-to-metal transition so-called Mott transition. As the temperature of VO2 film increases above a critical temperature, the phase of some domains starts to switch from insulating monoclinic (M1) to metallic rutile (R) phase [6]. Then, the number of domains whose phase is metallic gradually increases as we further increase the temperature, leading to a gradual change in resistance instead of conventional abrupt switching. We fabricated a proof-of-concept device that has a nanowire heater on the top of a VO2 gap with an electrical insulator in-between. With this device structure, the temperature of the VO2 film can be controlled precisely by adjusting the current flow through the heater. As a result, the resistance of the VO2 gap can be gradually decreased as we increase the amount of current flow through the heater, which emulates the output characteristics of ReLU function. More importantly, this Mott ReLU device can generate output voltage enough to drive another synaptic array without additional drivers. It enables direct stacking of multiple network layers with minimal peripheral circuits, which is essential for large-scale neural network hardware implementations.

  To investigate the advantages of using Mott ReLU device, we benchmarked the Mott ReLU device against CMOS implementations of ReLU activation function. In comparison to equivalent CMOS implementations, Mott ReLU devices occupy 1000× smaller area and consume much less energy. Moreover, when we implement a large-scale neural network for CIFAR-10 image classification with Mott ReLU devices, it requires a 100-1000× smaller area and consumes 10-100× less energy in comparison to the CMOS implementations. We also demonstrated successful convolutional edge detection operations with Mott ReLU devices and a nonvolatile memory device array in hardware. Our results show that Mott ReLU device is a promising solution toward large-scale, highly parallel, and energy-efficient in-memory computing systems for neural networks.

  For more information, please see our recent publication in Nature Nanotechnology and press release from University of California San Diego:


[1] Silver, David, et al. "Mastering the game of Go with deep neural networks and tree search." Nature 529.7587 (2016): 484-489.

[2] Schrittwieser, Julian, et al. "Mastering atari, go, chess and shogi by planning with a learned model." Nature 588.7839 (2020): 604-609.

[3] Zidan, Mohammed A., John Paul Strachan, and Wei D. Lu. "The future of electronics based on memristive systems." Nature Electronics 1.1 (2018): 22-29.

[4] Wong, H-S. Philip, et al. "Metal–oxide RRAM." Proceedings of the IEEE 100.6 (2012): 1951-1970.

[5] Yang, Tien-Ju, and Vivienne Sze. "Design considerations for efficient deep neural networks on processing-in-memory accelerators." 2019 IEEE International Electron Devices Meeting (IEDM). IEEE, 2019.

[6] Qazilbash, Mumtaz M., et al. "Mott transition in VO2 revealed by infrared spectroscopy and nano-imaging." Science 318.5857 (2007): 1750-1753.

Sangheon Oh

PhD Candidate, University of California San Diego