A High-performance RNS LSTM block

Vasilis Sakellariou, Vassilis Paliourasy, Ioannis Kouretasy, Hani Saleh, Thanos Stouraitis

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

3 Scopus citations

Abstract

The Residue Number System (RNS) has been proposed as an alternative to conventional binary representations for use in AI hardware accelerators. While it has been successfully utilized in applications targeting Convolutional Neural Networks (CNNs), its usage in other network models such as Recurrent Neural Networks (RNNs) has been set back due to the difficulty of implementing more complex activations functions like tanh and sigmoid (sigma) in the RNS domain. In this paper, we seek to extend its usage in such models, and in particular LSTM networks, by providing efficient RNS implementations of the activation functions. To this aim, we derive improved accuracy piecewise linear approximations of the tanh and sigma functions using the minimax approach and propose a fully RNS-based hardware realization. We show that our approximations can effectively mitigate accuracy degradation in LSTM networks compared to naive approximations, while the RNS LSTM block can be up to 40% more efficient in terms of performance per area unit compared to a binary counterpart, when used in high performance-targeted accelerators.

Original languageBritish English
Title of host publicationIEEE International Symposium on Circuits and Systems, ISCAS 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1264-1268
Number of pages5
ISBN (Electronic)9781665484855
DOIs
StatePublished - 2022
Event2022 IEEE International Symposium on Circuits and Systems, ISCAS 2022 - Austin, United States
Duration: 27 May 20221 Jun 2022

Publication series

NameProceedings - IEEE International Symposium on Circuits and Systems
Volume2022-May
ISSN (Print)0271-4310

Conference

Conference2022 IEEE International Symposium on Circuits and Systems, ISCAS 2022
Country/TerritoryUnited States
CityAustin
Period27/05/221/06/22

Keywords

  • AI Hardware Accelerator
  • LSTM
  • Residue Number System (RNS)

Fingerprint

Dive into the research topics of 'A High-performance RNS LSTM block'. Together they form a unique fingerprint.

Cite this