Research PapersNo Access

Incorporating external knowledge for image captioning using CNN and LSTM

Himanshu Sharma

http://orcid.org/0000-0001-5040-1759

Department of Computer Engineering and Applications, GLA University Mathura, Uttar Pradesh 281406, India

E-mail Address: himanshu.sharma@gla.ac.in

Corresponding author.

Search for more papers by this author

and

Anand Singh Jalal

Department of Computer Engineering and Applications, GLA University Mathura, Uttar Pradesh 281406, India

E-mail Address: asjalal@gla.ac.in

Search for more papers by this author

https://doi.org/10.1142/S0217984920503157Cited by:37 (Source: Crossref)

Abstract

Image captioning is a multidisciplinary artificial intelligence (AI) research task that has captures the interest of both image and natural language processing experts. Image captioning is a complex problem as it sometimes requires accessing the information that may not be directly visualized in a given scene. It possibly will require common sense interpretation or the detailed knowledge about the object present in image. In this paper, we have given a method that utilizes both visual and external knowledge from knowledge bases such as ConceptNet for better description the images. We demonstrated the usefulness of the method on two publicly available datasets; Flickr8k and Flickr30k.The results explain that the proposed model outperforms the state-of-the art approaches for generating image captions. At last, we will talk about possible future prospects in image captioning.

Keywords:

References

1. D. Bahdanau, K. Cho and Y. Bengio, Neural machine translation by jointly learning to align and translate (2014), arXiv preprint, arXiv:1409.0473. Google Scholar
2. K. Cho, B. Van Merrienboer, C. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, Learning phrase representations using RNN encoder–decoder for statistical machine translation, in Proc. 2014 Conf. Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Doha, Qatar, 2014, pp. 1724–1734. Google Scholar
3. I. Sutskever, O. Vinyals and Q. V. Le, Adv. Neural Inf. Process. Syst. 2 (2014) 3104. Google Scholar
4. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, Going deeper with convolutions, in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 7–12 June 2015 Boston, MA, USA, IEEE, 2015, pp. 1–9. Google Scholar
5. H. Liu and P. Singh, BT Technol. J. 22 (2004) 211. Web of Science, Google Scholar
6. C. Rashtchian, P. Young, M. Hodosh and J. Hockenmaier, Collecting image annotations using Amazons Mechanical Turk, in Proc. NAACL HLT 2010 Workshop on Creating Speech and language Data with Amazon’s Mechanical Turk, Association for Computational Linguistics, 2010, pp. 139–147. Google Scholar
7. P. Young, A. Lai, M. Hodosh and J. Hockenmaier, Trans. Assoc. Comput. Linguist. 2 (2014) 67. Google Scholar
8. J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang and A. Yuille, Deep captioning with multimodal recurrent neural networks (m-RNN), in Int. Conf. Learning Representations (ICLR), 2015. Google Scholar
9. Y. Pathak, K. V. Arya and S. Tiwari, Multimed. Tools Appl. 78 (2019) 1473, https://doi.org/10.1007/s11042-018-6155-6. Web of Science, Google Scholar
10. X. Jia, E. Gavves, B. Fernando and T. Tuytelaars, Guiding the long-short term memory model for image caption generation, in Proc. IEEE Int. Conf. Computer Vision, 7–13 December 2015, Santiago, Chile, IEEE, 2015, pp. 2407–2415. Google Scholar
11. K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel and Y. Bengio, Show, attend and tell: Neural image caption generation with visual attention, in Int. Conf. Machine Learning, 2015, pp. 2048–2057. Google Scholar
12. Q. Wu, C. Shen, P. Wang, A. Dick and A. van den Hengel, in IEEE Trans. Pattern Anal. Mach. Intell. 40 (2018) 1367. Web of Science, Google Scholar
13. J. Jin, K. Fu, R. Cui, F. Sha and C. Zhang, Aligning where to see and what to tell: image caption with region-based attention and scene factorization (2015), arXiv preprint, arXiv:1506.06272. Google Scholar
14. A. Karpathy and F.-F. Li, Deep visual-semantic alignments for generating image descriptions, in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Vol. 39(4) (IEEE, 2015), pp. 3128–3137. Google Scholar
15. R. Kiros, R. Salakhutdinov and R. Zemel, Multimodal neural language models, in Int. Conf. Machine Learning, 2014, pp. 595–603. Google Scholar
16. O. Vinyals, A. Toshev, S. Bengio and D. Erhan, in Show and tell: A neural image caption generator, in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 7–12 June 2015, Boston, MA, USA, 2015, pp. 3156–3164. Google Scholar
17. X. He et al., Pattern Recogn. Lett. 119 (2019) 229, https://doi.org/10.1016/j.patrec.2017.10.018. Web of Science, ADS, Google Scholar
18. P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould and L. Zhang, Bottom-up and top-down attention for image captioning and visual question answering, in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 8–23 June 2018, Salt Lake City, UT, USA, IEEE, 2018, pp. 6077–6086. Google Scholar
19. J. Aneja, A. Deshpande and A. Schwing, Convolutional image captioning, in Computer Vision and Pattern Recognition, 2018. Google Scholar
20. Q. Wang and A. B. Chan, CNN+CNN: Convolutional decoders for image captioning, CoRR, arXiv:abs/1805.09019, 2018. Google Scholar
21. J. Donahue, L. Hendricks, S. Guadarrama, M. Rohrbach and S. Venugopalan, Long-term recurrent convolutional networks for visual recognition and description, in IEEE Conf. Computer Vision and Pattern Recognition, Vol. 39(4) (IEEE, 2017), pp. 2625–2634. Google Scholar
22. K. Fu, J. Jin, R. Cui, F. Sha and C. Zhang, IEEE Trans. Pattern Anal. Mach. Intell. 39(12) (2017) 2321. Web of Science, Google Scholar
23. H. Fang, S. Gupta, F. Iandola and R. Srivastava, From captions to visual concepts and back, in IEEE Conf. Computer Vision and Pattern Recognition, IEEE, 2015, pp. 1473–1482. Google Scholar
24. X. Xiao, L. Wang, K. Ding, S. Xiang and C. Pan, IEEE Trans. Multimedia 21(11) (2019) 2942. https://doi.org/10.1109/TMM. 2019.2915033 Web of Science, Google Scholar
25. R. Xiang and F. Wu, Mod. Phys. Lett. B 32(34n36) (2018) 1840085. Link, Web of Science, Google Scholar
26. J. Zakraoui, S. Elloumi, J. M. Alja’am and S. B. Yahia, IEEE Access 7 (2019) 18772. https://doi.org/10.1109/ACCESS.2019.2896713 Web of Science, Google Scholar
27. M. Andrecut and M. K. Ali, Mod. Phys. Lett. B 15(1) (2001) 11. Link, Web of Science, ADS, Google Scholar
28. Z. Yuan, X. Li and Q. Wang, IEEE Access 8 (2020) 2608. https://doi.org/10.1109/ACCESS.2019.2962195 Web of Science, Google Scholar
29. L. Zheng, H. Shi and M. Gu, Mod. Phys. Lett. B 31(19–21) (2017) 1740044. Link, Web of Science, ADS, Google Scholar
30. S. Wang, L. Lan, X. Zhang, G. Dong and Z. Luo, IEEE Access 7 (2019) 66680. https://doi.org/10.1109/ACCESS.2019.2917979 Web of Science, Google Scholar
31. V. D. Singh, V. Kumar and M. Kaur, Eur. J. Clin. Microbiol. Infect. Dis. (2020). Google Scholar
32. M. Kaur, H. K. Gianey, D. Singh and M. Sabharwal, Mod. Phys. Lett. B 33(5) (2019) 1950022. Link, Web of Science, ADS, Google Scholar
33. M. Kaur and D. Singh, Cluster Comput. (2019) 1, https://doi.org/10.1007/s10586-019-02999-x. Google Scholar
34. H. S. Pannu, D. Singh and A. K. Malhi, CLEAN–Soil Air Water 46(5) (2018) 1700162. Web of Science, Google Scholar
35. H. S. Pannu, D. Singh and A. K. Malhi, Neural Comput. Appl. 31 (2019) 2195. Web of Science, Google Scholar
36. M. Kaur and V. Kumar, Int. J. Bifurcat. Chaos 28(11) (2018) 1850132. Link, Web of Science, Google Scholar
37. A. Gupta, D. Singh and M. Kaur, J. Amb. Intel. Hum. Comput. 11(3) (2020) 1309. Web of Science, Google Scholar
38. M. Kaur, D. Singh, K. Sun and U. Rawat, Future Gener. Comp. Syst. 107 (2020) 333. Web of Science, Google Scholar
39. M. Kaur, D. Singh and R. S. Uppal, IET Image Process. 14(6) (2020) 1015, https://doi.org/10.1049/iet-ipr.2019.0587. Web of Science, Google Scholar
40. M. Kaur, V. Kumar and L. Li, Neural Comput. Appl. 31(11) (2019) 7975. Web of Science, Google Scholar
41. D. Singh, V. Kumar and M. Kaur, Appl. Intell. 49(12) (2019) 4276. Web of Science, Google Scholar
42. P. K. Shukla, P. K. Shukla, P. Sharma, P. Rawat, J. Samar, R. Moriwal and M. Kaur, IET Syst. Biol. (2020), https://doi.org/10.1049/iet-syb.2019.0116. Google Scholar
43. D. Singh and V. Kumar, Sci. China Inform. Sci. 62(7) (2019) 79101. Web of Science, Google Scholar
44. M. Kaur, D. Singh, V. Kumar and K. Sun, Inf. Sci. 521 (2020) 326. Web of Science, Google Scholar
45. W. Shi, X. Zhang, X. Zou and W. Han, Mod. Phys. Lett. B 31(19–21) (2017) 1740096. Link, Web of Science, ADS, Google Scholar
46. K. Simonyan and A. Zisserman, in Int. Conf. on Learning Representations $(I C L R)$ $(I C L R)$ , 2015. Google Scholar
47. S. Hochreiter and J. Schmidhuber, Neural Comput. 9(8) (1997) 1735. Web of Science, Google Scholar
48. D. Singh and V. Kumar, Arch. Comput. Methods Eng. 26 (2019) 1395. Web of Science, Google Scholar
49. G. Qi, H. Wang, M. Haner, C. Weng, S. Chen and Z. Zhu, CAAI Trans. Intel. Technol. 4(2) (2019) 80. Google Scholar
50. X. Xue, J. Lu and J. Chen, AAI Trans. Intel. Technol. 4(3) (2019) 135. Google Scholar
51. H. S. Basavegowda and G. Dagnew, CAAI Trans. Intel. Technol. 5(1) (2020) 22. Google Scholar