Security certification of AI-based System in Multi-Agent Context

Research Theme

Multi-Agent Systems and Data Intelligence

This project is also supervised by Dr Sasa Radomirovic.

Aim

Develop a framework for the certification of AI-based systems for safety and security.

Objectives 

1. Formalise existing methods for assessing AI-based systems to identify building blocks for the framework.
2. Develop analytic methods combining compositional reasoning and machine learning pipelines to evaluate AI-based systems.
3. Apply the approach to the context of remote attestation in multi-agent systems.

Description

This project targets the fundamental question of the certification of AI-based data-systems for security and safety. This has been identified as a crucial open question by the 2023 AI Safety Summit and its recent follow-ups. Challenges posed by novel AI models require to re-think the analysis and certification of systems. By getting inspiration from the analysis of security and safety of software systems, this project combines knowledge and practice of machine learning and AI, and formal approaches of compositional reasoning for verification. The project is to conduct experiments targeting the problem of forming coalitions of agents in a multi-agent system where some of the agents do not belong or have been compromised. A solution requires remote attestation, whereby an agent proves that they are uncompromised (aside from authenticating themselves). The project contributes to Dstl’s secure by design adoption by providing a means to construct and evaluate secure AI-based systems.

Bengio et al. 2025. The Singapore Consensus on Global AI Safety Research Priorities. https://doi.org/10.48550/arXiv.2506.20702 
Dalrymple et al. 2024. Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems. https://doi.org/10.48550/arXiv.2405.06624 
Dstl. 2025. Secure by Design Problem Book. https://www.gov.uk/government/publications/secure-by-design-problem-book 

Minimum entry qualification - an Honours degree at 2:1 or above (or International equivalent) in a relevant science or engineering discipline, possibly supported by an MSc Degree. 

Further information on English language requirements for EU/Overseas applicants.

Home fee rate and stipend available for this position.