A Regularization Approach to Maximize Common Sub-Expressions in Neural Network Weights

This paper modifies the training of artificial neural networks in order to derive weights the binary expression of which is composed of a limited set of sub-expressions only, placed in specified positions. The benefit of the proposed technique is the substantial complexity reduction achieved. The proposed method can be applied as a post-processing step on pre-trained models, further expanding its impact. The main concept is the use of an introduced regularization function that promotes specific sub-expressions which, in turn, are shown to improve the performance of common sub-expression sharing techniques, reducing area, time and power requirements for inference. Synthesis results reported here, quantify the impact of the proposed method onto hardware implementations and demonstrate substantial area, delay and power improvements, over prior art. In certain cases a ×4 area reduction is achieved, combined with corresponding reduction in delay and power dissipation.