TIGARS

Developing an assurance strategy should be a key part of the overall design approach and integrated into the overall lifecycle. The assurance approach should be commensurate with the different risks and consistent across them, e.g., by adopting an outcome-based risk informed approach.

Novel assurance approaches (e.g., articulated using CAE) exclusive to ML and AI-based systems should be developed to identify areas to focus on and establish how they impact both the system and its assurance. It can help define and evaluate the reasoning and evidence needed.

Key claims should address the high-level functional and ethical principles such as those from the EU Expert Group report [9] and the Sherpa project [10]. These principles can be used to shape and define system or service level properties.

An assurance case for autonomous systems should at a minimum address the points below:

• what the system is and in what environment and ecosystem it will operate in
• how much trust in a system is needed, considering interdependencies and systemic risks
• whether it is sufficiently trustworthy to be initially deployed
• whether it will continue to be trustworthy in the face of environmental changes, threat evolution and failure

Structured argumentation for safety cases (and more generally assurance cases) needs more emphasis on reasoning and evidence, if the cases are to be sufficiently robust and acceptable. We have characterised a new CAE based assurance framework to achieve this, which would utilise evidence extracted from V&V, defence in depth, and diversity techniques.

Autonomous vehicles (AV’s) deployed as part of an ecology of systems that deliver services (e.g., mobility service), must appropriately define resilience and safety requirements.

To develop the dependability or resilience of a service, discussions of requirements and assurance should start from a service level, not from systems or components level. Discussions should include vehicle capabilities, infrastructure sensors, cloud systems, etc. Resilience requirements should also be derived at service level, and then assigned to each system or component. The system should have high-level safety, security and resilience requirements. A systems theoretic approach (e.g., using Systems Theoretic Process Analysis - STPA) combined with an impact of variability using Functional Resonance Analysis Method (FRAM), can be useful to addressing these requirements.

For AVs to be accepted socially, stakeholders need to have confidence, before they are deployed, in how they are going to adapt to changes post-deployment. This is particularly important when considering security requirements, as AVs are deployed in threat-filled environments that keep changing.

The future behaviours of AVs should be assured systematically through Open Systems Dependability (OSD) deployed on the system’s lifecycle.

Policy makers and regulators should enable and promote adoption of standardised OSD by AV manufacturers, operators and users. They play a key role in ensuring such collaboration is possible across legitimate AVs. Sharing of information, including assurance, is a major concern. Appropriate policies can incentivise AV manufacturers and operators to participate in a transparent evaluation and assurance regime which in turn strengthens users’ confidence in Robotics and Autonomous Systems (RASs’) future behaviours.

It is crucial to strategize the use of V&V (and defence in depth and diversity techniques) through the lens of an assurance approach, in particular, CAE, to identify the role of such methods and how they complement other approaches. We provide recommendations regarding their use below:

Formal Verification

ML-specific properties such as pointwise robustness not only fail to note real-world examples, but also how state-of-the-art verification techniques can be applied to real-time systems. With regard to the safety assurance of an autonomous system, pointwise robustness fails to support or provide evidence for system robustness.

We thus recommend:

Creation of relevant safety specifications unique to ML algorithms, with corresponding mathematical frameworks. The noted specifications must contribute to the assurance of an AI system, specifically, the context of an assurance case (i.e., CAE). Some ML algorithms (e.g., vision) may be intrinsically unverifiable against the properties which are of interest to the safety of an autonomous vehicle, however, other properties can in principle be formulated for other types of ML systems (e.g., planning) in autonomous vehicles.

Collaboration between ML and verification researchers resulting in deep learning systems that are more amenable to verification. Novel formal verification techniques are needed which can address the newly defined specifications.

Static Analysis

The ML lifecycle relies heavily on data processed in a complex chain of libraries and tools traditionally implemented, often in Python. It has been demonstrated that implementation in these systems may propagate and affect the accuracy and functionality of the ML algorithm itself. We have demonstrated that static analysis tools can be used to build confidence in supporting systems. However, the verification of existing ML software infrastructure may pose particular challenges.

We thus recommend:

Creation of novel formally-based static analysis techniques addressing Python, and more generally, dynamically typed languages, given that they are not currently available. Formal methods can have a strong role in ensuring provenance of training and data processing.

Organisations should consider rewriting any deployed safety critical software in a verifiable language if the appropriate analysis tools for Python are unavailable.

Organisations must understand the extent to which existing integrity static analysis tools can contribute to the confidence of the development of ML algorithms. The complexities arising from choice of implementation language, e.g., issues from using C or C++, should be well understood.

Simulation

The roles of the different simulation variants should be specified and justified, and confidence in the simulation environment needs to be established. This may include confidence in the modelled behaviour of the tested system, as well as confidence in the software running the simulation.
Attempts should be made to make the tests as repeatable as possible, however, if this is not possible the impact in confidence on the test results must be considered.

Adjustments in system behaviour may be needed to accommodate the simulation environments and these will need to be justified so that test evidence can be used in the overall assurance case.

The use of diversity to improve reliability and safety is a sound principle. In particular it should be used to achieve higher dependability of safety mechanisms. The stakeholders for a mobility service or AVs should undertake a review of defence in depth and define a diversity and defence in depth strategy balancing the advantages of diversity with possible increases in complexity and attack surface.

Diversity should be considered within the construction of a system’s architecture to reduce the trust needed in a single ML component. Independence of failures should not be assumed and failure correlation should be considered based where possible on experimental data. An architectural approach which limits reliance on sub-components of the system that need to be highly trusted (e.g., ML algorithms) should be taken.

Safety monitor architectures should be considered to reduce the trust needed in ML components as they monitor both the state of the environment and the AV. Where feasible, they can be used to gain performance and safety benefits of deploying complex ML components, whilst mitigating the risks of using such technologies.

The use of diversity to improve reliability and safety is a sound principle. In particular it should be used to achieve higher dependability of safety mechanisms. The stakeholders for a mobility service or AVs should undertake a review of defence in depth and define a diversity and defence in depth strategy balancing the advantages of diversity with possible increases in complexity and attack surface.

Diversity should be considered within the construction of a system’s architecture to reduce the trust needed in a single ML component. Independence of failures should not be assumed and failure correlation should be considered based where possible on experimental data. An architectural approach which limits reliance on sub-components of the system that need to be highly trusted (e.g., ML algorithms) should be taken.

Safety monitor architectures should be considered to reduce the trust needed in ML components as they monitor both the state of the environment and the AV. Where feasible, they can be used to gain performance and safety benefits of deploying complex ML components, whilst mitigating the risks of using such technologies.

Security-informed safety (SIS) should be addressed at all stages of the innovation cycle from conceptualisation, experimentation, and prototyping through to production.  A security-informed hazard analysis should be undertaken during development. The hazard analysis should be reviewed periodically during operation or when a safety related component has been updated or if additional threat and vulnerability information has been identified.

The UK PAS 11281 can be used to systematically consider security through addressing: security policy, organisation and culture; security-aware development process; maintaining effective defences; incident management; and safe and secure design, all contributing to a safe and secure world. The PAS can be applied progressively during the innovation lifecycle of an AV or RAS, and adapted to provide a project specific implementation.

Security-informed safety cases are still novel, and the experience of developing and integrating security issues into the safety analysis should be captured. In the industry as a whole, more training and expertise for SIS analysis is required, as many decisions rely on expert judgement. Although methodology that has been developed in other sectors can also be applied to AVs, AI and ML based technologies will provide novel security challenges that must additionally be addressed.

Duplication of standardisation work on similar topics should be reduced to a minimum. Efforts to prevent duplication have been ongoing in the international standardisation community, but we have observed that AV and RAS relevant topics often have duplicate standards, which may not be aligned. An example of possible duplication is in risk management.

An authoritative and introductory guideline covering necessary knowledge for AVs and RASs should be developed for new entrants to this arena. Particularly, guidelines should include surveys on foundational standards of the field. Many IT companies are entering into the market without the experience of the traditional manufacturers. The current lack of such overall guidelines can lead to IT stakeholders to overly concentrate on their strength within a particular area without essential knowledge of AI/ML or safety. The recommended guideline would help ensure that innovative technologies and traditional engineering and assurance practices are aligned.