International Journal of Software Engineering and Knowledge Engineering

Existing software product line (SPL) engineering testing approaches generally provide positive testing that validates the SPL’s functionality. Negative testing is commonly neglected. This research aims to unify behavioral and feature models of an SPL, enable testing before and after variability binding for domain-centric and product-centric testing, and combine positive and negative testing for a holistic testing view. This study suggests behavioral modeling with event sequence graphs (ESGs). This heterogeneous modeling strategy supports bottom-up domain testing and top-down product testing with the feature model. This new feature-oriented ESG test creation method generates shorter test sequences than the original ESG optimum test sequences. Statechart and original ESG test-generating methods are compared. Positive testing findings are similar. The Statechart technique generated 12 test cases with 59 events, whereas the ESG technique created six test cases with 60 events. The ESG technique generated 205 negative test cases with 858 events with the Test Suite Designer tool. However, the Conformiq Designer tool for the Statechart technique does not have a negative test case generation capability. It is shown that the proposed ESG-based holistic approach confirms not only the desirable (positive) properties but also the undesirable (negative) ones. As an additional research, the traditional ESG test-generating approach is compared to the new feature-oriented method on six SPLs of different sizes and features. Our case study results show that the traditional ESG test generation approach demonstrated higher positive test generation scores compare to the proposed feature-oriented test generation approach. However, our proposed feature-oriented test generation approach is capable of generating shorter test sequences, which could be beneficial for reducing the execution time of test cases compared to traditional ESG approach. Finally, our case study has also shown that regardless of the test generation approach, there has been found no significant difference between the Bottom-up and Top-down test strategies with respect to their positive test generation scores.

Software architecture security design is a key stage in developing business-oriented system, such as business-critical system, ICT system and AI system. Many typical accidents also remind us that the security of software architecture even plays a more important role than the code security in most software systems. However, there are very few researches which focus on the security of software architecture. Especially, we don’t find a systematic and feasible quantitative method to evaluate the security of software architecture. To fill this gap, we launched a research project focusing on software architecture security since 2019 and try to provide a series of quantitative methods for evaluating the security of software architecture. In this paper, a business-oriented quantitative method was proposed to evaluate the security of software architecture from the view of business-critical security insurance. In our method, both business dependency graph (BDG) and multi-hierarchical dependency graph (MHDG) are defined and constructed first for describing businesses and their relationships, and system components and their relationships; then, attack points and business key-points are identified and labeled on BDG and MHDG; and further, all potential attack paths and effective attack paths in the system based on MHDG are detected; and finally, a set of security indicators is defined and used to evaluate the security of software architecture quantitatively. For more effective experiments, we use software architectures with different styles and different evolution versions. Experimental results show that our method can reflect effectively the security characteristics of software architecture with different styles, can find the security changes of different architecture evolution versions for the same system, and can perform quantitative evaluation of software architecture security efficiently.

Neural networks are extensively employed in safety-critical systems. However, these critical systems incorporating neural networks continue to pose risks due to the presence of adversarial examples. Although the security of neural networks can be enhanced by verification, verifying neural networks is an NP-hard problem, making the application of verification algorithms to large-scale neural networks a challenging task. For this reason, we propose NNTBFV, a framework that utilizes the principles of Testing-Based Formal Verification (TBFV) to simplify neural networks and verify the simplified networks. Unlike conventional neural network pruning techniques, this approach is based on specifications, with the goal of deriving approximate execution paths under given preconditions. To mitigate the potential issue of unverifiable conditions due to overly broad preconditions, we also propose a precondition partition method. Empirical evidence shows that as the range of preconditions narrows, the size of the execution paths also reduces accordingly. The execution path generated by NNTBFV is still a neural network, so it can be verified by verification tools. In response to the results from the verification tool, we provide a theoretical method for analysis.

We evaluate the effectiveness of NNTBFV on the ACAS Xu model project, choosing Verification-based and Random-based neural network simplification algorithms as the baselines for NNTBFV. Experiment results show that NNTBFV can effectively approximate the baseline in terms of simplification capability, and it surpasses the efficiency of the random-based method.

With the flourish of the Internet of Things (IoT), the group communication Constrained Application Protocol (CoAP) emerged at the historic moment, enabling homogeneous devices with constrained computing ability to communicate with ease. CoAP is widely used in transportation, health care and many other aspects. Hence, it is prominent to propose a flexible and efficient architecture for usage in such scenarios and study the data security and consistency of the architecture from the perspective of formal methods. In this paper, we extend the group communication CoAP model to the enhanced group communication CoAP by the introduction of smart gateways and binding to new security suites. We make further improvements to increase the scalability and flexibility of the architecture, making it more applicable in a healthcare scenario or smart home scenario. And we adopt process algebra Communicating Sequential Processes (CSP) with real-time extension to model the enhanced group communication CoAP. We use model checker PAT to verify eight properties of our model on an abstract level, including four basic properties and four security properties. We also conduct a simulation on the local machine for validation of the above properties on a more detailed level. Despite some simplifications of physical properties, both results of the verification and simulation show that the proposed architecture can satisfy those requirements and demonstrate a good possibility of being securely put into service.

As a fundamental component of software documentation, code comments could help developers comprehend and maintain programs. Several datasets of method header comments have been proposed in previous studies for machine learning-based code comment generation. As part of code comments, inline code comments are also crucial for code understanding activities. However, unlike method header comments written in a standard format and describing the whole method code, inline comments are often written in arbitrary formats by developers due to timelines pressures and different aspects of code snippets in the method are described. Currently, there is no large-scale dataset used for inline comments generation considering these. Hence, this naturally inspires us to explore whether we can construct a dataset to foster machine learning research that not only performs fine-grained noise-cleaning but conducts a taxonomy of inline comments. To this end, we first collect inline comments and code snippets from 8000 Java projects on GitHub. Then, we conduct a manual review to obtain heuristic rules, which could be used to clean the data noise in a fine-grained manner. As a result, we construct a large-scale benchmark dataset named ICG with 5,740,770 pairs of inline comments and code snippets. We then build a comprehensive taxonomy and conduct a statistical and manual analysis to explore the performances of different categories of inline comments, such as helpfulness in code understanding. After that, we provide and compare several baseline models to automatically generate inline comments, such as CodeBERT, to enhance the usability of the benchmark for researchers. The availability of our benchmark and baselines can help develop and validate new inline comment generation methods, which would also further facilitate code understanding activities.

Search-based Test Case Generation (TCG) for web applications suffers from unstable performance and suboptimal test suite problems due to diversity loss. However, previous diversity metrics mainly only focus on client-side models or server-side code, which are prone to low robustness and poor generalization in practical applications. We propose a diversity-driven TCG method DTester, which can maximize behavior exploration and minimize the test suite size while covering more server-side vulnerable paths. Three diversity metrics (i.e. phenotypic coupling, intent coupling and competitiveness) are proposed to measure the underlying relationship between test cases from user behavior, code logic and test execution history. Moreover, a 3-dimensional weight graph is designed to model association among metrics, which provides fine-grained guidance for the genetic algorithm to generate diverse test cases from the client-side behavior model. Our empirical evaluation on five web applications shows that DTester can efficiently and robustly generate better test suites than the state-of-the-art TCG method. The maximum improvement is $85\%$, $60\%$, $716\%$ and $88\%$ in efficiency, test suite size, diversity and robustness.