A Survey and Comparative Analysis of Number Systems for Deep Neural Networks

Deep neural networks (DNNs) are indispensable in various artificial intelligence (AI) applications. However, their inherent complexity presents significant challenges, particularly when deploying them on resource-constrained devices. To overcome these hurdles, academia and industry are actively seeking ways to accelerate and optimize DNN implementations. A significant area of research revolves around discovering more effective methods to represent the enormous data volumes processed by DNNs. Traditional number systems (NSs) have proven nonoptimal for this task, prompting extensive exploration into alternative and bespoke systems for DNNs. This survey aims to comprehensively discuss various NSs utilized to efficiently represent DNN data. These systems are categorized mainly based on their impact on DNN performance and hardware implementation. This survey offers an overview of these categorized NSs and delves into different subsystems within each, outlining their effect on DNN performance and hardware design. Furthermore, these systems are compared quantitatively and qualitatively concerning their expected quantization error, memory utilization, and computational requirements. This survey also emphasizes the challenges linked with each system and the diverse proposed solutions to address them. Insights into the utilization of these NSs for sophisticated DNNs are also presented in this survey. Readers will acquire a deeper understanding of the importance of efficient NSs for DNNs, explore commonly used systems, comprehend the tradeoffs between these systems, delve into design considerations influencing their impact on DNN performance, and discover recent trends and potential research avenues in this field.