Software Engineering of Fault Tolerant Systems

The following section are included:

• PREFACE

• International Advisory Committee

• CONTENTS

The following section are included:

• Motivations for the Book

• Dependability and Fault Tolerance

• Defining Software Engineering
  • If Software Fails, This May Cost Millions of Dollars and Harm People
  • How to Make Good Software

• Fault Tolerance Engineering: from Requirements to Code
  • Requirements Engineering and Fault Tolerance
  • Software Architecture and Fault Tolerance
  • Low-level Design and Fault Tolerance

• Verification and Validation of Fault Tolerant Systems
  • Model Checking
  • Theorem Provers
  • Constraint Solvers
  • Testing
  • UML-based approaches to modelling and validating dependable systems

• Languages and Frameworks
  • Programming Languages Perspectives
    • Exception Handling
    • Atomic Actions
    • Reflection and Aspect-Orientation
  • Frameworks for Fault Tolerance
  • Advanced Frameworks for Fault Tolerance

• Contribution of this Book to the Topic

• Acknowledgments

• References

Database replication has gained a lot of attention in the last few years since databases are often the bottleneck of modern information systems. Database replication is often implemented at the middleware level. However, there is an important tradeoff in terms of scalability and performance if the database is dealt with as a black box. In this paper, we advocate for a gray box approach in which some minimal functionality of the database is exposed through a reflective interface enabling performant and scalable database replication at the middleware level. Reflection is the adequate paradigm to separate the regular functionality from the replication logic. However, traditional full-fledged reflection is too expensive. For this reason, the paper focuses on the exploration of some lightweight reflective interfaces that can combine the architectonic advantages of reflection with good performance and scalability. The paper also evaluates thoroughly the cost of the different proposed reflective mechanisms and its impact on the performance.

Late detection of new types of faults often results in the evolution of fault-tolerance requirements while developers have already created design artifacts. Thus, the reuse of an existing design in the development of a fault-tolerant version thereof has the potential to reduce the overall development costs. Moreover, the automation of such a reuse yields a fault-tolerant design that is correct by construction, given that the existing design is correct. To facilitate such an automation, we present an approach, where we add three levels of fault-tolerance, namely failsafe, nonmasking, and masking, to functional designs represented as state machines. Intuitively, failsafe fault-tolerance requires that safety specification is met even in the presence of faults. In the presence of faults, nonmasking fault-tolerance guarantees recovery to states from where safety and liveness specifications are satisfied. Masking fault-tolerance stipulates that (i) recovery is provided to states from where safety and liveness specifications are met, and (ii) safety specification is satisfied during such a recovery. Specifically, we present sound and complete deterministic algorithms for automated addition of (failsafe/nonmasking/masking) fault-tolerance to the functional design of concurrent programs. These polynomial-time algorithms are especially useful in model-driven development of fault-tolerant systems, where models are automatically checked and modified. We also discuss (1) the effect of distribution and safety specification model on the complexity of adding fault-tolerance, and (2) the impact of the proposed algorithms on the addition of multitolerance.

Replication is a well-known technique to enhance dependability of distributed systems. A plethora of replication techniques for distributed object, file, and database systems has been proposed in the past decades. However, replication in service-oriented systems is still in its infancy. Thus, in this chapter we address replication on a conceptual level and contribute to the understanding of replication in service-oriented systems by (i) analyzing replication protocols regarding their suitability for service-oriented systems and by (ii) identifying commonalities in existing service replication middleware architectures and deriving a system architecture pattern for replication of services. Our main conclusion is that neither replication protocols nor replication middleware architectures need to be re-invented from scratch for service-oriented systems. Thus, we recommend software and service engineers to take a look at the well-established replication algorithms and architectures for distributed object, database, and file systems and apply these techniques in service-oriented systems. However, there are some subtle differences that have to be taken into consideration.

Everyday thanks to technological advances software of embedded systems is more complex and its increasing utilization for critical applications, makes necessary the development and later validation of fault tolerant embedded systems. Moreover, the increasing utilization of commercial off-the-shelf (COTS) components to develop such systems increases the necessity of verifying the behavior properties exhibited by each system component in presence of faults. To face this challenge, verification and validation techniques becomes essential to evaluate the fault tolerant properties of such embedded systems. But in order to obtain representative results of fault tolerance, the verification and validation processes have to be non-intrusive, mainly when evaluating fault tolerant real-time systems. Using the on-chip debugging capabilities available in most modern embedded systems it is possible to perform fault injection experiments in a non-intrusive manner and observe the system behavior in front of such faults. This paper argues how to use such on-chip debugging (OCD) facilities for verification and validation of fault tolerant real-time embedded systems constructed from COTS components integration.

This chapter presents two runtime error detection techniques for UML 2.0 statechart implementations. The first technique aims at detecting errors caused by model refinement faults (introduced in early phases of the development) by proposing a temporal logic language to be used for defining and checking temporal correctness criteria on statecharts. The second solution is a watchdog structure aiming at detection of errors caused by implementation faults. The solutions can detect a subset of errors emerging from operational fault as well.

Fault-tolerant communication is a crucial point in building distributed safety-critical real-time systems, as they are used today e.g. in the automotive and avionics domain. To argue about the timing properties of a distributed system and to show the fault-tolerance of its communication, a predictable timing of the system is needed. This can be solved using the time-triggered paradigm. In accordance with this paradigm, a time-triggered communication protocol, FlexRay, and an operating system OSEKtime with corresponding communication layer FTCom for the fault-tolerant communication were introduced by the FlexRay Consortium and OSEK/VDX respectively.

In this chapter we present the formal specifications of FlexRay and FTCom that allow us not only to argue about their properties in a precise, formal manner and to infer the dependences between their properties, but also to prove the correctness of the implementation formally.

Programming languages provide exception handling mechanisms to structure fault tolerant activities within software systems. However, the use of exceptions at this low level of abstraction can be error-prone and complex, potentially leading to new programming errors. In this chapter, we present a model-driven framework to support the iterative development of reliable software systems. This framework is comprised of UML-based modeling notations and a transformation engine that supports the automated generation of exception management features for a software system. It leverages domain specific exception modeling languages, patterns, modeling tools and framework libraries. The feasibility of this approach is demonstrated through the development of a case study business application, known as Project Tracker.

In software engineering appropriately developed reusable components may significantly reduce the design cost, shorten the time to the market and reduce the maintenance effort. For those reasons more and more attention has been paid to the component-based framework recently. In this chapter, to improve availability of each individual service instance in a component-based system, we propose a runtime configurable fault management mechanism (FMM) which utilizes model-based failure detection and rule-based repair. When more than one repair action is possible, FMM picks one that incurs the best tradeoff between the success rate, the cost of repair and the post-repair reliability measure. Furthermore, in order to gradually improve the quality of fault tolerance, FMM is designed to be able to accumulate knowledge and adapts its capability accordingly.

In this paper we present our recent work extending the applicability of a toolset for the fast prototyping, verification and quantitative validation of dependable distributed systems and algorithms.

Neko is a framework and a communication platform that allows rapid prototyping of distributed algorithms; the same implementation of an algorithm can be exercised both on top of real and simulated networks, allowing simulative and experimental qualitative and, using the NekoStat extension, quantitative analyses. The core of the Neko framework is written in Java (J2SE), being thus highly portable; however it requires the translation into Java of existing algorithms, written in other languages, that one wants to analyze. The Neko package contains also utilities that help users to configure campaigns of experiments; these supports are really useful, but they are made as Perl scripts and Unix C code and are thus directly usable only on Unix systems.

Our work aimed at extending the applicability of the tool was made towards two directions. On one side we included in the framework the utilities to allow a direct integration of existing C/C++ algorithms in Neko applications, avoiding the translation into Java that may be error prone and that may also fail in correctly represent some low level details of the algorithms. Since most of the running distributed algorithms are written in C/C++, allowing a direct analysis of C/C++ existing legacy distributed algorithms, we widely extend the applicability of Neko and improve the faithfulness of analyses performed. The paper describes the extensions made and illustrates the use of the tool on an algorithm whose Java translation does not have the original behavior. On the other side, we made a pure Java code (J2SE, Java 2 Standard Edition) porting of all utilities helping Neko users in the definition of simulative/experimental campaigns, obtaining thus a complete J2SE version of Neko.