Research PaperNo Access

Deep Learning-Based Image Geolocation for Travel Recommendation via Multi-Task Learning

Fangfang Gu

College of Economics and Management, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, P. R. China

E-mail Address: gff_0817@163.com

Search for more papers by this author

Keshen Jiang

College of Economics and Management, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, P. R. China

E-mail Address: jiangksh@126.com

Search for more papers by this author

Xiaoyi Hu

College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210023, P. R. China

E-mail Address: 1320017314@njupt.edu.cn

Search for more papers by this author

, and

Jie Yang

College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210023, P. R. China

E-mail Address: jyang@njupt.edu.cn

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S0218126622501274Cited by:3 (Source: Crossref)

Abstract

Localizing images by visual information is a very challenging task in image-based travel recommendations. Travelers take a large number of pictures every day and share them on social networks (Facebook, Sina Weibo, Yelp, etc.). Many of these images are associated with the location where they are taken. But for images that do not associate with geographic location information, how to estimate where they are taken? With the rapid development of social media, the increasing number of shared geographic-labeled images brings an opportunity to address this problem. Using geographic-labeled images to estimate the location of unlabeled images is a popular approach. In this paper, we propose an image geographic location estimation model via multi-task learning (GLML). It combines the classification task and retrieval task to calculate the similarity between the query image and dataset images. Additionally, it fuses multi-global features through multiple global pooling techniques to enhance feature extraction. Each part of the proposed GLML model is flexible and extensible. Experiments on seven public datasets show the effectiveness of the proposed model.

This paper was recommended by Regional Editor Takuro Sato.

Keywords:

References

1. T. Weyand, I. Kostrikov and J. Philbin , Planet-photo geolocation with convolutional neural networks, Proc. Eur. Conf. Comput. Vis. (Springer, 2016), pp. 37–55. Crossref, Google Scholar
2. P. H. Seo et al., Cplanet: Enhancing image geolocalization by combinatorial partitioning of maps, Proc. Eur. Conf. Comput. Vis. (Springer, 2018), pp. 536–551. Crossref, Google Scholar
3. J. Hays and A. A. Efros , IM2GPS: Estimating geographic information from a single image, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2008), pp. 1–8. Crossref, Google Scholar
4. Y. Li, D. J. Crandall and D. P. Huttenlocher , Landmark classification in large-scale image collections, Proc. IEEE 12th Int. Conf. Comput. Vis. (IEEE, 2009), pp. 1957–1964. Google Scholar
5. N. Vo, N. Jacobs and J. Hays , Revisiting im2gps in the deep learning era, Proc. IEEE Int. Conf. Comput. Vis. (IEEE, 2017), pp. 2640–2649. Crossref, Google Scholar
6. A. Torii et al., 24/7 place recognition by view synthesis, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2015), pp. 1808–1817. Crossref, Google Scholar
7. J. Li et al., GPS estimation for places of interest from social users’ uploaded photo, IEEE Trans. Multimedia 15 (2013) 2058–2071. Crossref, Web of Science, Google Scholar
8. T. Quack, B. Leibe and L. Van Gool , World-scale mining of objects and events from community photo collections, Proc. 2008 Int. Conf. Content-based Image and Video Retrieval. (CIVR, 2008), pp. 47–56. Crossref, Google Scholar
9. K. Vishal, C. V. Jawahar and V. Chari , Accurate localization by fusing images and GPS signals, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops (IEEE, 2015), pp. 17–24. Crossref, Google Scholar
10. J. Hays and A. A. Efros , Large-Scale Image Geolocalization, Multimodal Location Estimation of Videos and Images (Springer, 2015), pp. 41–62. Crossref, Google Scholar
11. R. Arandjelovic et al., NetVLAD: CNN architecture for weakly supervised place recognition, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2016), pp. 5297–5307. Crossref, Google Scholar
12. H. J. Kim, E. Dunn and J. M. Frahm , Learned contextual feature reweighting for image geo-localization, Proc. 2017 IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2017), pp. 3251–3260. Crossref, Google Scholar
13. G. Baatz et al., Handling urban location recognition as a 2d homothetic problem, Proc. Eur. Conf. Comput. Vis. (Springer, Berlin, Heidelberg, 2010), pp. 266–279. Crossref, Google Scholar
14. S. Cao and N. Snavely , Graph-based discriminative learning for location recognition, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2013), pp. 700–707. Crossref, Google Scholar
15. D. M. Chen et al., City-scale landmark identification on mobile devices, in Proc. CVPR 2011 (IEEE, 2011), pp. 737–744. Crossref, Google Scholar
16. H. J. Kim, E. Dunn and J. M. Frahm , Predicting good features for image geo-localization using per-bundle vlad, Proc. IEEE Int. Conf. Comput. Vis. (IEEE, 2015), pp. 1170–1178. Crossref, Google Scholar
17. J. Knopp, J. Sivic and T. Pajdla , Avoiding confusing features in place recognition, Proc. Eur. Conf. Comput. Vis. (Springer, Berlin, Heidelberg, 2010), pp. 748–761. Crossref, Google Scholar
18. J. Philbin et al., Object retrieval with large vocabularies and fast spatial matching, Proc. 2007 IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2007), pp. 1–8. Crossref, Google Scholar
19. G. Schindler, M. Brown and R. Szeliski , City-scale location recognition, Proc. 2007 IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2007), pp. 1–7. Crossref, Google Scholar
20. A. R. Zamir and M. Shah , Accurate image localization based on google maps street view, Proce. Eur. Conf. Comput. Vis. (Springer, Berlin, Heidelberg, 2010), pp. 255–268. Crossref, Google Scholar
21. A. R. Zamir, M. Shah , Image geo-localization based on multiplenearest neighbor feature matching usinggeneralized graphs, IEEE Trans. Pattern Anal. Mach. Intell. 36 (2014) 1546–1558. Crossref, Web of Science, Google Scholar
22. H. Noh et al., Large-scale image retrieval with attentive deep local features, Proc. IEEE Int. Conf. Comput. Vis. (IEEE, 2017), pp. 3456–3465. Crossref, Google Scholar
23. T. Y. Lin et al., Learning deep representations for ground-to-aerial geolocalization, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2015), pp. 5007–5015. Crossref, Google Scholar
24. N. N. Vo and J. Hays , Localizing and orienting street views using overhead imagery, Proc. Eur. Conf. Comput. Vis. (Springer, Cham, 2016), pp. 494–509. Crossref, Google Scholar
25. M. Bansal, K. Daniilidis and H. Sawhney , Ultrawide Baseline Facade Matching for Geo-Localization, Large-Scale Visual Geo-Localization (Springer, Cham, 2016), pp. 77–98. Crossref, Google Scholar
26. S. Workman, R. Souvenir and N. Jacobs , Wide-area image geolocalization with aerial reference imagery, Proc. IEEE Int. Conf. Comput. Vis. (IEEE, 2015), pp. 3961–3969. Crossref, Google Scholar
27. K. Simonyan, and A. Zisserman , Very deep convolutional networks for large-scale image recognition, Proc. ICLR (2015), arXiv:1409.1556v6. Google Scholar
28. S. Ioffe and C. Szegedy , Batch normalization: Accelerating deep network training by reducing internal covariate shif, Proc. ICML (2015), pp. 448–456. Google Scholar
29. K. He et al., Deep residual learning for image recognition, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2016), pp. 770–778. Crossref, Google Scholar
30. M. Lin, Q. Chen and S. Yan , Network in network, in Proc. ICLR (Poster) (2014), arXiv:1312.4400v3. Google Scholar
31. F. Radenović, G. Tolias and O. Chum , Fine-tuning CNN image retrieval with no human annotation, IEEE Trans. Pattern Anal. Mach. Intell. 41 (2019) 1655–1668. Crossref, Web of Science, Google Scholar
32. A. Babenko and V. Lempitsky , Aggregating deep convolutional features for image retrieval, Proc. ICCV (2015), arXiv:1510.07493. Google Scholar
33. A. S. Razavian et al., Visual instance retrieval with deep convolutional networks, ITE Trans. Media Technol. Appl. 4 (2016) 251–258. Crossref, Google Scholar
34. G. Tolias, R. Sicre and H. Jégou , Particular object retrieval with integral max-pooling of CNN activations, Proc. ICLR (2016), arXiv:1511.05879. Google Scholar
35. A. Hermans, L. Beyer and B. Leibe , In defense of the triplet loss for person re-identification, Proc. ICLR (2017), arXiv:1703.07737. Google Scholar
36. J. Philbin et al., Lost in quantization: Improving particular object retrieval in large scale image databases, Proc. 2008 IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2008), pp. 1–8. Crossref, Google Scholar
37. A. Radenović et al., Revisiting oxford and paris: Large-scale image retrieval benchmarking, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (IEEE, 2018), pp. 5706–5715. Crossref, Google Scholar
38. Y. Kalantidis, C. Mellina and S. Osindero , Cross-dimensional weighting for aggregated deep convolutional features, Proc. Eur. Conf. Comput. Vis. (Springer, Cham, 2016), pp. 685–701. Crossref, Google Scholar
39. X. Shi and X. Qian , Exploring spatial and channel contribution for object based image retrieval, Knowl. Based Syst. 186 (2019) 104955. Crossref, Web of Science, Google Scholar