Deep LearningNo Access

A Layer-Wise Ensemble Technique for Binary Neural Network

Department of Computer Science and Engineering, Fukuoka Institute of Technology, 3-30-1 Wajiro-higashi, Higashi-ku, Fukuoka 811-0295, Japan

E-mail Address: bd18101@bene.fit.ac.jp

Corresponding author.

Search for more papers by this author

and

Hiroyuki Yamauchi

Department of Computer Science and Engineering, Fukuoka Institute of Technology, 3-30-1 Wajiro-higashi, Higashi-ku, Fukuoka 811-0295, Japan

E-mail Address: yamauchi@fit.ac.jp

Search for more papers by this author

https://doi.org/10.1142/S021800142152011XCited by:0 (Source: Crossref)

Abstract

Binary neural networks (BNNs) have drawn much attention because of the most promising techniques to meet the desired memory footprint and inference speed requirements. However, they still suffer from the severe intrinsic instability of the error convergence, resulting in increase in prediction error and its standard deviation, which is mostly caused by the inherently poor representation with only two possible values of $-$ 1 and $+$ 1. In this work, we have proposed a cost-aware layer-wise ensemble method to address the above issue without incurring any excessive costs, which is characterized by (1) layer-wise bagging and (2) cost-aware layer selection for the bagging. One of the experimental results has shown that the proposed method reduces the error and its standard deviation by 15% and 54% on CIFAR-10, respectively, compared to the BNN serving as a baseline. This paper demonstrated and discussed such error reduction and stability performance with high versatility based on the comparison results under the various cases of combinations of the network base model with the proposed and the state-of-the-art prior techniques while changing the network sizes and datasets of CIFAR-10, SVHN, and MNIST for the evaluation.

Keywords:

References

1. Z. Baozhou, P. Hofstee, J. Lee and Z. Al-Ars, SoFAr: Shortcut-based fractal architectures for binary convolutional neural networks, arXiv:2009.05317. Google Scholar
2. L. Breiman, Bagging predictors, Mach. Learn. 24(2) (1996) 123–140. Crossref, Web of Science, Google Scholar
3. L. Breiman, Bias, variance, and arcing classifiers, Tech. Rep. 460, Statistics Department, University of California, Berkeley (1996). Google Scholar
4. A. Bulat and G. Tzimiropoulos, XNOR- $Net + +$ : Improved binary neural networks, arXiv:1909.13863. Google Scholar
5. M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv and Y. Bengio, Binarized neural networks: Training deep neural networks with weights and activations constrained to $+ 1$ or $- 1$ , arXiv:1602.02830. Google Scholar
6. Y. Freund and R. E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, J. Comput. Syst. Sci. 55(1) (1997) 119–139. Crossref, Web of Science, Google Scholar
7. J. Friedman et al., Additive logistic regression: A statistical view of boosting (with discussion and a rejoinder by the authors), Ann. Stat. 28(2) (2000) 337–407. Crossref, Web of Science, Google Scholar
8. Y. Guo, A survey on methods and theories of quantized neural networks, arXiv:1808.04752. Google Scholar
9. S. Han, Z. Meng, A.-S. Khan and Y. Tong, Incremental boosting convolutional neural network for facial action unit recognition, in Proceedings of the 30th International Conference on Neural Information Processing Systems (Curran Associates Inc., Barcelona, Spain, 2016), pp. 109–117. Google Scholar
10. F. Huang, J. Ash, J. Langford and R. Schapire, Learning deep ResNet blocks sequentially using boosting theory, arXiv:1706.04964. Google Scholar
11. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv and Y. Bengio, Binarized neural networks, in Proceedings of the 30th International Conference on Neural Information Processing Systems (Curran Associates Inc., Barcelona, Spain, 2016), pp. 4107–4115. Google Scholar
12. A. Krizhevsky, Learning multiple layers of features from tiny images (2009). Google Scholar
13. X. Lin, C. Zhao and W. Pan, Towards accurate binary convolutional neural network, in Proceedings of the 31st International Conference on Neural Information Processing Systems (Curran Associates Inc., Long Beach, California, USA, 2017), pp. 345–353. Google Scholar
14. J. Liu, D. Wen, D. Wang, W. Tao, T.-W. Chen, K. Osa and M. Kato, QuantNet: Learning to quantize by learning within fully differentiable framework, arXiv:2009.04626. Google Scholar
15. B. Martinez, J. Yang, A. Bulat and G. Tzimiropoulos, Training binary neural networks with real-to-binary convolutions, arXiv:2003.11535. Google Scholar
16. P. Merolla, R. Appuswamy, J. Arthur, S. K. Esser and D. Modha, Deep neural networks are robust to weight binarization and other non-linear distortions, arXiv:1606.01981. Google Scholar
17. D. Miyashita, E. H. Lee and B. Murmann, Convolutional neural networks using logarithmic data representation, arXiv:1603.01025. Google Scholar
18. M. Moghimi, S. J. Belongie, M. J. Saberian, J. Yang, N. Vasconcelos and L.-J. Li, Boosted convolutional neural networks, in Proc. British Machine Vision Conf. (BMVC), Vol. 5 (BMVA Press, 2016), p. 6. Crossref, Google Scholar
19. N. C. Oza and S. Russell, Online Ensemble Learning (University of California, Berkeley, 2001). Google Scholar
20. A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga and A. Lerer, Automatic differentiation in PyTorch (2017). Google Scholar
21. M. Rastegari, V. Ordonez, J. Redmon and A. Farhadi, XNOR-Net: ImageNet classification using binary convolutional neural networks, in 14th European Conference Computer Vision (ECCV 2016) (Springer, Amsterdam, The Netherlands, 2016), pp. 525–542. Crossref, Google Scholar
22. R. E. Schapire and Y. Freund, Boosting: Foundations and algorithms, Kybernetes 42 (2013) 164–166. Crossref, Google Scholar
23. T. Simons and D.-J. Lee, A review of binarized neural networks, Electronics 8(6) (2019) 661. Crossref, Web of Science, Google Scholar
24. Z. H. Zhou, Ensemble Methods: Foundations and Algorithms (CRC Press, 2012). Crossref, Google Scholar
25. S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen and Y. Zou, DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients, arXiv:1606.06160. Google Scholar
26. S. Zhu, X. Dong and H. Su, Binary ensemble neural network: More bits per network or more networks per bit? in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), pp. 4923–4932, https://doi.org/10.1109/CVPR.2019.00506. Crossref, Google Scholar