by Richard Dorrance, Deepak Dasalukunte, Hechen Wang, Renzhi Liu, Brent Carlton
Abstract:
In recent years, Neural Networks (NNs) have achieved tremendous success in a variety of fields, such as computer vision, natural language processing, speech recognition, autonomous driving, and healthcare [1] –[4]. However, conventional NNs rely heavily on large, labeled training datasets, which can lead to overfitting and overconfident decision making (especially when faced with unfamiliar, out-of-distribution inputs) [1] –[4]. Unlike conventional NNs, the weights of Bayesian Neural Network (BNNs) are represented by probability distributions, providing a mathematical framework to quantify the uncertainties in a model’s final prediction [3]. These uncertainty estimates allow BNNs to mitigate overfitting issues, enable training with smaller datasets, and help increase overall model accuracy through the implicit use of stochastic rounding [5]. Figure 1 shows an example with a BNN version of LeNet-5 [6], where weights of the network are represented by Gaussian distributions. The ambiguous digit is classified as a ‘5’ and ‘3’ with probabilities of 80% and 20%, respectively. However, these uncertainty estimates come at great computational cost, as multiple forward inference passes are required to generate the necessary posterior distributions. As such, BNNs require not only efficient, high-performance multiply-accumulation (MAC), but an efficient Gaussian Random Number Generator (GRNG) with high-quality statistics as well.
Reference:
R. Dorrance, D. Dasalukunte, H. Wang, R. Liu, B. Carlton, "Energy Efficient BNN Accelerator using CiM and a Time-Interleaved Hadamard Digital GRNG in 22nm CMOS," in 2022 IEEE Asian Solid-State Circuits Conference (A-SSCC), pp. 2–4, November 2022.
Bibtex Entry:
@INPROCEEDINGS{Dorrance2022:ASSCC,
author = {Dorrance, Richard and Dasalukunte, Deepak and Wang, Hechen and Liu, Renzhi and Carlton, Brent},
title = {{Energy Efficient BNN Accelerator using CiM and a Time-Interleaved Hadamard Digital GRNG in 22nm CMOS}},
booktitle = {2022 IEEE Asian Solid-State Circuits Conference (A-SSCC)},
year = {2022},
month = {November},
pages = {2--4},
doi = {10.1109/A-SSCC56115.2022.9980539},
abstract = {In recent years, Neural Networks (NNs) have achieved tremendous success in a variety of fields, such as computer vision, natural language processing, speech recognition, autonomous driving, and healthcare [1] –[4]. However, conventional NNs rely heavily on large, labeled training datasets, which can lead to overfitting and overconfident decision making (especially when faced with unfamiliar, out-of-distribution inputs) [1] –[4]. Unlike conventional NNs, the weights of Bayesian Neural Network (BNNs) are represented by probability distributions, providing a mathematical framework to quantify the uncertainties in a model’s final prediction [3]. These uncertainty estimates allow BNNs to mitigate overfitting issues, enable training with smaller datasets, and help increase overall model accuracy through the implicit use of stochastic rounding [5]. Figure 1 shows an example with a BNN version of LeNet-5 [6], where weights of the network are represented by Gaussian distributions. The ambiguous digit is classified as a ‘5’ and ‘3’ with probabilities of 80% and 20%, respectively. However, these uncertainty estimates come at great computational cost, as multiple forward inference passes are required to generate the necessary posterior distributions. As such, BNNs require not only efficient, high-performance multiply-accumulation (MAC), but an efficient Gaussian Random Number Generator (GRNG) with high-quality statistics as well.},
url = {https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9980539}
}