Special Issue on Artificial Intelligence and Knowledge Engineering; Guest Editor: Ajay BansalNo Access

Feature Fusion and Augmentation Based on Manifold Ranking for Image Classification

Department of Statistics, Applied Mathematics and Computing (DEMAC), São Paulo State University (UNESP), Rio Claro, 13506-900, São Paulo, Brazil

E-mail Address: nessahelena@gmail.com

Corresponding author.

Search for more papers by this author

Lucas Pascotti Valem

https://orcid.org/0000-0002-3833-9072

Department of Statistics, Applied Mathematics and Computing (DEMAC), São Paulo State University (UNESP), Rio Claro, 13506-900, São Paulo, Brazil

E-mail Address: lucas.valem@unesp.br

Search for more papers by this author

Gustavo Rosseto Leticio

https://orcid.org/0009-0008-3715-8991

Department of Statistics, Applied Mathematics and Computing (DEMAC), São Paulo State University (UNESP), Rio Claro, 13506-900, São Paulo, Brazil

E-mail Address: gustavo.leticio@unesp.br

Search for more papers by this author

, and

Daniel Carlos Guimarães Pedronette

https://orcid.org/0000-0002-2867-4838

Department of Statistics, Applied Mathematics and Computing (DEMAC), São Paulo State University (UNESP), Rio Claro, 13506-900, São Paulo, Brazil

E-mail Address: daniel.pedronette@unesp.br

Search for more papers by this author

https://doi.org/10.1142/S1793351X24440033Cited by:0 (Source: Crossref)

Abstract

Despite the great advances in the field of image classification, the association of ideal approaches that can bring improved results, considering different datasets, is still an open challenge. In this work, a novel approach is presented, based on a combination of compared strategies: feature extraction for early fusion; rankings based on manifold learning for late fusion; and feature augmentation applied in a long short-term memory (LSTM) algorithm. The proposed method aims to investigate the effect of feature fusion (early fusion) and ranking fusion (late fusion) in the final results of image classification. The experimental results showed that the proposed strategies improved the accuracy of results in different tested datasets (such as CIFAR10, Stanford Dogs, Linnaeus 5, Flowers 102, and Flowers 17) using a fusion of features from three convolutional neural networks (CNNs) (ResNet152, VGG16, and DPN92) and its respective generated rankings. The results indicated significant improvements and showed the potential of the approach proposed for image classification.

Keywords:

References

1. L. D. Nguyen, D. Lin, Z. Lin and J. Cao , Deep CNNs for microscopic image classification by exploiting transfer learning and feature concatenation, in IEEE Int. Symp. Circuits and Systems (IEEE, 2018), pp. 1–5. Crossref, Google Scholar
2. S. Patil and S. Bhosale , Assessing feature selection techniques for machine learning models using cardiac dataset, in IEEE 5th Int. Conf. Artificial Intelligence and Knowledge Engineering (IEEE, 2022), pp. 126–130. Crossref, Google Scholar
3. M. K. M. Rabby, R. B. Eshun, S. Belkasim and A. K. Islam , Epileptic seizure detection using EEG signal based LSTM models, in IEEE 4th Int. Conf. Artificial Intelligence and Knowledge Engineering (IEEE, 2021), pp. 131–132. Crossref, Google Scholar
4. M. K. M. Rabby, R. B. Eshun, S. Belkasim and A. K. Islam , Time-frequency based eeg motor imagery signal classification with deep learning networks, in IEEE 4th Int. Conf. Artificial Intelligence and Knowledge Engineering (IEEE, 2021), pp. 133–134. Crossref, Google Scholar
5. G. Kumar and P. K. Bhatia , A detailed review of feature extraction in image processing systems, in 4th Int. Conf. Advanced Computing n Communication Technologies (IEEE, 2014), pp. 5–12. Crossref, Google Scholar
6. J. Wu , Introduction to convolutional neural networks, Natl. Key Lab Nov. Softw. Technol. 5 (23) (2017) 495. Google Scholar
7. J. Zeng, X. Ma and K. Zhou , Enhancing attention-based LSTM with position context for aspect-level sentiment classification, IEEE Access 7 (2019) 20462–20471. Crossref, Web of Science, Google Scholar
8. D. Lauss, F. Eibensteiner and P. Petz , A deep learning based hand gesture recognition on a low-power microcontroller using IMU sensors, in 21st IEEE Int. Conf. Machine Learning and Applications (IEEE, 2022), pp. 733–736. Crossref, Google Scholar
9. D. C. G. Pedronette, L. P. Valem and R. S. Torres , A BFS-tree of ranking references for unsupervised manifold learning, Pattern Recognit. 111 (2021) 107666–107679. Crossref, Web of Science, Google Scholar
10. D. C. G. Pedronette, L. P. Valem, J. Almeida and R. S. Torres , Multimedia retrieval through unsupervised hypergraph-based manifold ranking, IEEE Trans. Image Process. 28(12) (2019) 5824–5838. Crossref, Web of Science, Google Scholar
11. B. Fernando, E. Fromont, D. Muselet and M. Sebban , Discriminative feature fusion for image classification, in IEEE Conf. Computer Vision and Pattern Recognition (IEEE, 2012), pp. 3434–3441. Crossref, Google Scholar
12. Z. Zhuang, Z. Yang, A. N. J. Raj, C. Wei, P. Jin and S. Zhuang , Breast ultrasound tumor image classification using image decomposition and fusion based on adaptive multi-model spatial feature fusion, Comput. Methods Programs Biomed. 208 (2021) 106221. Crossref, Web of Science, Google Scholar
13. S. Susan and J. Malhotra , Learning image by-parts using early and late fusion of auto-encoder features, Multimedia Tools Appl. 80(19) (2021) 29601–29615. Crossref, Web of Science, Google Scholar
14. Q.-S. Sun, S.-G. Zeng, Y. Liu, P.-A. Heng and D.-S. Xia , A new method of feature fusion and its application in image recognition, Pattern Recognit. 38 (2005) 2437–2448. Crossref, Web of Science, Google Scholar
15. K. He, X. Zhang, S. Ren and J. Sun , Deep residual learning for image recognition, in Proc. IEEE Conf. Computer Vision Pattern Recognition (IEEE, 2016), pp. 770–778. Crossref, Google Scholar
16. K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, preprint (2014), arXiv:1409.1556. Google Scholar
17. F. Pedregosa et al., Scikit-learn: Machine learning in Python, J. Mach. Learn. Res. 12 (2011) 2825–2830. Web of Science, Google Scholar
18. Y.-H. Tu, J. Du, Q. Wang, X. Bao, L.-R. Dai and C.-H. Lee , An information fusion framework with multi-channel feature concatenation and multi-perspective system combination for the deep-learning-based robust recognition of microphone array speech, Comput. Speech Lang. 46 (2017) 517–534. Crossref, Web of Science, Google Scholar
19. U. G. Mangai, S. Samanta, S. Das and P. R. Chowdhury , A survey of decision fusion and feature fusion strategies for pattern classification, IETE Tech. Rev. 27(4) (2010) 293–307. Crossref, Web of Science, Google Scholar
20. J. Yang, K. Yu, Y. Gong and T. Huang , Linear spatial pyramid matching using sparse coding for image classification, in IEEE Conf. Computer Vision and Pattern Recognition (IEEE, 2009), pp. 1794–1801. Google Scholar
21. Y. Tang, L. Zhang, F. Min and J. He , Multiscale deep feature learning for human activity recognition using wearable sensors, IEEE Trans. Ind. Electron. 70(2) (2022) 2106–2116. Crossref, Google Scholar
22. S. Rabba, M. Kyan, L. Gao, A. Quddus, A. S. Zandi and L. Guan , Discriminative robust head-pose and gaze estimation using kernel-DMCCA features fusion, Int. J. Semant. Comput. 14(01) (2020) 107–135. Link, Google Scholar
23. D. Shah, K. J. Han and S. S. Narayanan , Robust multimodal person recognition using low-complexity audio-visual feature fusion approaches, Int. J. Semant. Comput. 4(02) (2010) 155–179. Link, Google Scholar
24. L. P. Valem and D. C. G. Pedronette , Unsupervised selective rank fusion for image retrieval tasks, Neurocomputing 377 (2020) 182–199. Crossref, Web of Science, Google Scholar
25. M. K. Panda, B. N. Subudhi, T. Veerakumar and V. Jakhetiya , Modified ResNet-152 network with hybrid pyramidal pooling for local change detection, IEEE Trans. Artif. Intell. 5(4) (2023) 1599–1612. Crossref, Google Scholar
26. R. Volpi, P. Morerio, S. Savarese and V. Murino , Adversarial feature augmentation for unsupervised domain adaptation, in Proc. IEEE Conf. Computer Vision and Pattern Recognition (IEEE, 2018), pp. 5495–5504. Crossref, Google Scholar
27. T. DeVries and G. W. Taylor , Dataset augmentation in feature space, Workshop track, Int. Conf. Learning Representations, 2017, pp. 1–12. Google Scholar
28. V. H. Pereira-Ferrero, L. P. Valem and D. C. G. Pedronette , Feature augmentation based on manifold ranking and LSTM for image classification, Expert Syst. Appl. 213 (2023) 118995. Crossref, Web of Science, Google Scholar
29. F. A. Gers, J. Schmidhuber and F. Cummins , Learning to forget: Continual prediction with LSTM, Neural Comput. 12(10) (2000) 2451–2471. Crossref, Web of Science, Google Scholar
30. S. Hochreiter and J. Schmidhuber , Long short-term memory, Neural Comput. 9(8) (1997) 1735–1780. Crossref, Web of Science, Google Scholar
31. A. Farzad, H. Mashayekhi and H. Hassanpour , A comparative performance analysis of different activation functions in LSTM networks for classification, Neural Comput. Appl. 31 (2019) 2507–2521. Crossref, Web of Science, Google Scholar
32. G. Chaladze and L. Kalatozishvili, Linnaeus 5 dataset for machine learning (2017), http://chaladze.com/l5/. Google Scholar
33. M.-E. Nilsback and A. Zisserman , A visual vocabulary for flower classification, in IEEE Computer Society Conf. Computer Vision and Pattern Recognition, Vol. 2 (IEEE, 2006), pp. 1447–1454. Crossref, Google Scholar
34. M.-E. Nilsback and A. Zisserman , Automated flower classification over a large number of classes, in 2008 Sixth Indian Conf. Computer Vision, Graphics & Image Processing (IEEE, 2008), pp. 722–729. Crossref, Google Scholar
35. A. Khosla, N. Jayadevaprakash, B. Yao and L. Fei-Fei , Novel dataset for fine-grained image categorization, in 1st Workshop on Fine-Grained Visual Categorization, IEEE Conf. Computer Vision and Pattern Recognition, Vol. 2 (IEEE, 2011), pp. 1–2. Google Scholar
36. A. Krizhevsky et al., Learning multiple layers of features from tiny images, University of Toronto, Toronto (2009), pp. 1–58. Google Scholar
37. H. Perez and J. H. Tah , Improving the accuracy of convolutional neural networks by identifying and removing outlier images in datasets using t-SNE, Mathematics 8(5) (2020) 662. Crossref, Web of Science, Google Scholar