No Access

Automated Visual Testing of Application User Interfaces Using Static Analysis of Screenshots

Šarūnas Packevičius

Software Engineering Department, Kaunas University of Technology, Studentų 50-406, Kaunas 51368, Lithuania

E-mail Address: sarunas.packevicius@ktu.edu

Search for more papers by this author

Greta Rudžionienė

Software Engineering Department, Kaunas University of Technology, Studentų 50-406, Kaunas 51368, Lithuania

E-mail Address: greta.rudzioniene@ktu.edu

Search for more papers by this author

, and

Eduardas Bareiša

Software Engineering Department, Kaunas University of Technology, Studentų 50-406, Kaunas 51368, Lithuania

E-mail Address: eduardas.bareisa@ktu.edu

Search for more papers by this author

https://doi.org/10.1142/S0218194021500042Cited by:1 (Source: Crossref)

Abstract

Mobile and web applications must operate and be displayed correctly on many different devices and browsers. The visual testing of web or mobile applications is usually a manual process that requires a significant amount of testing time, meaning that applications are tested only on a few devices. It is then assumed that the applications will be displayed correctly on other compatible or similar devices. This paper presents an automated visual testing method for user interfaces. The main contributions of this paper are a classification scheme for visual defects of user interfaces and the definition of an automatic visual testing method that tests applications on many different devices with varying hardware and software parameters. The method is based on an automated search for defects using heuristic and expected state prediction algorithms, which involves analyzing the resources used by applications and screenshots. The testing method works by executing applications on a full set of devices, taking a screenshot at every execution step, and analyzing each of these screenshots. The manual as well as automated testing approaches were validated on 781 of Android applications. The experimental results show that the proposed method has advantages over manual testing.

Keywords:

References

1. S. Wagner, A literature survey of the quality economics of defect-detection techniques, in Proc. 2006 ACM/IEEE Int. Symp. Empirical Software Engineering, 2006, pp. 194–203. Crossref, Google Scholar
2. T. Parr, Web application internationalization and localization in action, in Proc. 6th Int. Conf. Web Engineering, 2006, pp. 64–70. Crossref, Google Scholar
3. D. Amalfitano, A. R. Fasolino, P. Tramontana, S. De Carmine and A. M. Memon, Using GUI ripping for automated testing of Android applications, in Proc. 27th IEEE/ACM Int. Conf. Automated Software Engineering, 2012, pp. 258–261. Crossref, Google Scholar
4. S. Anand, M. Naik, M. J. Harrold and H. Yang, Automated concolic testing of smartphone apps, in Proc. ACM SIGSOFT 20th Int. Symp. Foundations of Software Engineering, 2012, pp. 1–11. Crossref, Google Scholar
5. A. N. Q. X. Atif Memon, Automating regression testing for evolving GUI software, J. Softw. Maint. Evolution: Res. Practice 17(1) (2005) 27–64. Crossref, Google Scholar
6. C. Hu and I. Neamtiu, Automating GUI testing for Android applications, in Proc. 6th Int. Workshop Automation of Software Test, 2011, pp. 77–83. Crossref, Google Scholar
7. L. Kanglin and W. Mengqi, Effective GUI Test Automation: Developing an Automated GUI Testing Tool (Sybex, Boston, 2005). Google Scholar
8. D. H. Nguyen, P. Strooper and J. G. Süß, Automated functionality testing through GUIs, in Proc. 33rd Australasian Conf. Computer Science, 2010, pp. 153–162. Google Scholar
9. T. Su, G. Meng, Y. Chen, K. Wu, W. Yang, Y. Yao, G. Pu, Y. Liu and Z. Su, Guided, stochastic model-based GUI testing of Android apps, in Proc. 2017 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 245–256. Crossref, Google Scholar
10. C.-F. Chen, M. Pistoia, C. Shi, P. Girolami, J. W. Ligman and Y. Wang, UI X-Ray: Interactive Mobile UI Testing Based on Computer Vision, in Proc. 22nd Int. Conf. Intelligent User Interfaces, 2017, pp. 245–255. Crossref, Google Scholar
11. V. Lelli, A. Blouin and B. Baudry, Classifying and qualifying GUI defects, in 2015 IEEE 8th Int. Conf. Software Testing, Verification and Validation, 2015, pp. 1–10. Crossref, Google Scholar
12. J. Johnson, GUI Bloopers 2.0: Common User Interface Design Don’ts and Dos (Morgan Kaufmann Publishers Inc., Burlington, MA 01803, USA, 2007). Google Scholar
13. K. Reinecke and A. Bernstein, Knowing what a user likes: A design science approach to interfaces that automatically adapt to culture, Mis Quart. 37(2) (2013) 427–453. Crossref, Web of Science, Google Scholar
14. Š. Packevičius, D. Barisas, A. Ušaniov, E. Guogis and E. Bareiša, Text semantics and layout defects detection in android apps using dynamic execution and screenshot analysis, in Information and Software Technologies, 2018, pp. 279–292. Crossref, Google Scholar
15. K. Moran, B. Li, C. Bernal-Cárdenas, D. Jelf and D. Poshyvanyk, Automated reporting of GUI design violations for mobile apps, in 40th Int. Conf. Software Engineering, 2018, 11 pages. Crossref, Google Scholar
16. Y.-M. Baek and D.-H. Bae, Automated model-based Android GUI testing using multi-level GUI comparison criteria, in Proc. 31st IEEE/ACM Int. Conf. Automated Software Engineering, 2016, pp. 238–249. Crossref, Google Scholar
17. T. Yeh, T.-H. Chang and R. C. Miller, Sikuli: Using GUI screenshots for search and automation, in Proc. 22nd Annual ACM Symp. User Interface Software and Technology, 2009, pp. 183–192. Crossref, Google Scholar
18. A. Méndez-Porras, J. Alfaro-Velásco, M. Jenkins and A. Martínez Porras, A user interaction bug analyzer based on image processing, CLEI Electronic Journal 19(2) (2016) 1–13. Google Scholar
19. T.-H. Chang, T. Yeh and R. C. Miller, GUI testing using computer vision, in Proc. SIGCHI Conf. Human Factors in Computing Systems, 2010, pp. 1535–1544. Crossref, Google Scholar
20. M. Ernst, Static and dynamic analysis: Synergy and duality, in ICSE’03 Int. Conf. Software Engineering, 2003, pp. 25–29. Google Scholar
21. N. Borges, Data flow oriented UI testing: Exploiting data flows and UI elements to test Android applications, in Proc. 26th ACM SIGSOFT Int. Symp. Software Testing and Analysis, 2017, pp. 432–435. Crossref, Google Scholar
22. Y. Li, Z. Yang, Y. Guo and X. Chen, Humanoid: A deep learning-based approach to automated black-box android app testing, 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), San Diego, CA, USA, 2019, pp. 1070–1073, https://doi.org/10.1109/ASE.2019.00104. Crossref, Google Scholar
23. Y. Li, Z. Yang, Y. Guo and X. Chen, DroidBot: A lightweight UI-guided test input generator for Android, in Proc. 39th Int. Conf. Software Engineering Companion, 2017, pp. 23–26. Google Scholar
24. D. Amalfitano, A. R. Fasolino, P. Tramontana and N. Amatucci, Considering context events in event-based testing of mobile applications, in 2013 IEEE 6th Int. Conf. Software Testing, Verification and Validation Workshops, 2013, pp. 126–133. Crossref, Google Scholar
25. T. Azim and I. Neamtiu, Targeted and depth-first exploration for systematic testing of android apps, SIGPLAN Not. 48(10) (2013) 641–660. Crossref, Google Scholar
26. W. Choi, G. Necula and K. Sen, Guided GUI testing of Android apps with minimal restart and approximate learning, SIGPLAN Not. 48(10) (2013) 623–640. Crossref, Google Scholar
27. S. R. Ganov, C. Killmar, S. Khurshid and D. E. Perry, Test generation for graphical user interfaces based on symbolic execution, in Proc. 3rd Int. Workshop Automation of Software Test, 2008, pp. 33–40. Crossref, Google Scholar
28. A. Machiry, R. Tahiliani and M. Naik, Dynodroid: An input generation system for Android apps, in Proc. 2013 9th Joint Meeting on Foundations of Software Engineering, 2013, pp. 224–234. Crossref, Google Scholar
29. Y. Xun and A. M. Memon, Generating event sequence-based test cases using gui runtime state feedback, IEEE Trans. Softw. Eng. 36(1) (2010) 81–95. Crossref, Web of Science, Google Scholar
30. T. Su, FSMdroid: Guided GUI testing of android apps, in Proc. 38th Int. Conf. Software Eng. Companion, 2016, pp. 689–691. Crossref, Google Scholar
31. X. Zeng, D. Li, W. Zheng, F. Xia, Y. Deng, W. Lam, W. Yang and T. Xie, Automated test input generation for Android: Are we really there yet in an industrial case?, in Proc. 2016 24th ACM SIGSOFT Int. Symp. Foundations of Software Engineering, 2016, pp. 987–992. Crossref, Google Scholar
32. C.-J. M. Liang, N. D. Lane, N. Brouwers, L. Zhang, B. F. Karlsson, H. Liu, Y. Liu, J. Tang, X. Shan, R. Chandra and F. Zhao, Caiipa: Automated large-scale mobile app testing through contextual fuzzing, in Proc. 20th Annual Int. Conf. Mobile Computing and Networking, 2014, pp. 519–530. Crossref, Google Scholar
33. J. A. Saddler and M. B. Cohen, EventFlowSlicer: A tool for generating realistic goal-driven GUI tests, in Proc. 32nd IEEE/ACM Int. Conf. Automated Software Engineering, 2017, pp. 955–960. Crossref, Google Scholar
34. B. Brykczynski, A survey of software inspection checklists, SIGSOFT Softw. Eng. Notes 24(1) (1999) 82. Crossref, Google Scholar
35. B. Caldwell, M. Cooper, L. G. Reid and G. Vanderheiden, Web content accessibility guidelines (WCAG) 2.0, 2008, https://www.w3.org/TR/WCAG20/. Google Scholar
36. A. Kay, Tesseract: An open-source optical character recognition engine, Linux J. 2007(159) (2007) 2. Google Scholar
37. T. Beltramelli, pix2code: Generating code from a graphical user interface screenshot, in Proc. ACM SIGCHI Symp. Engineering Interactive Computing Systems, 2018, pp. 3. Crossref, Google Scholar
38. P. Hinks and J. S. H. Lok, System and methods for translating software into localized versions, U.S. Patent US5678039A. Google Scholar
39. L. Frisén and M. Frisén, How good is normal visual acuity?, Albrecht von Graefes Arch. Klin. Ophthalmol. 215(3) (1981) 149–157. Crossref, Google Scholar
40. N. Oco and A. Borra, A grammar checker for Tagalog using language tool, in Proc. 9th Workshop Asian Language Resources, 2011, pp. 2–9. Google Scholar
41. T. R. Jordan, Presenting words without interior letters: Superiority over single letters and influence of postmask boundaries, J. Exp. Psychol.: Human Perception Performance 16(4) (1990) 893–909. Crossref, Web of Science, Google Scholar
42. A. M. Ayyal Awwad and W. Slany, Automated bidirectional languages localization testing for android apps with rich GUI, J. Mob. Inf. Syst. 2016 (2016) 13. Web of Science, Google Scholar