The outcome should contribute to:
- Standardisation of testing for dialogue systems.
- Enhanced clarity on the performances of dialogue systems for all stakeholders, including system developers, funders, and users.
- Community building at the European defence level.
- Trustworthy dialogue systems that enhance operational decision-making.
- Availability of databases to further develop dialogue systems.
Human-AI dialogue systems offer impressive results but are still prone to errors of various types. Moreover, there is no established metric to measure system performances. In order to ensure trustworthiness and steer progress, these systems should be submitted to common tests using shared data and clear metrics and protocols.
The goal of this call topic is thus to set up a testing environment and organise a technological challenge to evaluate the performances of such systems for defence use cases, including their abilities to manage classified information and to justify their answers. The challenge should be open to research teams supported through another call topic (EDF-2025-LS-RA-CHALLENGE-DIGIT-HAIDP) and possibly by other sources of funding. Representative defence users should be involved to contribute to the definition of the use cases and associated data, to test the demonstrators produced by the participating teams, and to provide feedback.
Scope:The proposals should address the organisation of a technological challenge on human-AI dialogue
...
