SAFEXPLAIN

Deep Learning (DL) techniques are at the heart of most future advanced software functions in Critical Autonomous AI-based Systems (CAIS), where they also represent a major competitive factor. Hence, the economic success of CAIS industries (e.g., automotive, space, railway) depends on their ability to design, implement, qualify, and certify DL-based software products under bounded effort/cost. However, there is a fundamental gap between Functional Safety (FUSA) requirements on CAIS and the nature of DL solutions. This gap stems from the development process of DL libraries and affects high-level concepts such as (1) explainability and traceability, (2) suitability for varying safety requirements, (3) FUSA-compliant implementations, and (4) real-time constraints. As a matter of fact, the data-dependent and stochastic nature of DL algorithms clashes with current FUSA practice, which instead builds on deterministic, verifiable, and pass/fail test-based software.

The safe use of AI in critical systems such as cars, trains and satellites

The SAFEXPLAIN project tackles the challenges by providing a novel and flexible approach to allow the certification – hence adoption – of DL-based solutions in CAIS building on (1) DL solutions that provide end-to-end traceability, with specific approaches to explain whether predictions can be trusted and strategies to reach (and prove) correct operation, in accordance to certification standards; (2) alternative and increasingly sophisticated design safety patterns for DL with varying requirements of criticality and fault tolerance; (3) DL library implementations that adhere to safety requirements; and (4) computing platform configurations, to regain determinism, and probabilistic timing analyses, to handle the remaining non-determinism.

Any system with safety requirements (transportation, industrial, medical, etc.).

Unleash the potential of AI for autonomous operation (e.g., autonomous driving) while preserving safety.

Some open-source libraries (with permissive libraries), and some software with commercial licenses. Software still being developed (early stages), so no explicit protection yet.

Automotive, space, railway, avionics, industries.