As autonomous systems like self-driving cars and planetary robots are more widely used, reacting to novel situations has become more important. In order to evaluate the agent’s detection ability, it is necessary to disentangle the performance from the intrinsic difficulty of novelties. A common type of novelties in the real world is based on an object, like a new vehicle type on the road.
A recent research paper on arXiv.org presents a qualitative physics-based method to quantify the difficulty of detecting novel objects with the same appearance but changed physical parameters.
The approach is applied to Angry Birds, where the world resembles real-world physics. The introduced novelties included pigs that fall down slower or stone blocks more difficult to destroy. An experiment with human players confirms that the proposed difficulty measure is in line with the detection difficulty for humans.
Detecting and responding to novel situations in open-world environments is a key capability of human cognition. Current artificial intelligence (AI) researchers strive to develop systems that can perform in open-world environments. Novelty detection is an important ability of such AI systems. In an open-world, novelties appear in various forms and the difficulty to detect them varies. Therefore, to accurately evaluate the detection capability of AI systems, it is necessary to investigate the difficulty to detect novelties. In this paper, we propose a qualitative physics-based method to quantify the difficulty of novelty detection focusing on open-world physical domains. We apply our method in a popular physics simulation game, Angry Birds. We conduct an experiment with human players with different novelties in Angry Birds to validate our method. Results indicate that the calculated difficulty values are in line with the detection difficulty of the human players.
Research paper: Pinto, V., Xue, C., Nagoda Gamage, C., and Renz, J., “The Difficulty of Novelty Detection in Open-World Physical Domains: An Application to Angry Birds”, 2021. Link: https://arxiv.org/abs/2106.08670