Curriculum
Computer Science and Innovation for Societal Challenges, XXXV series
Grant sponsor
Fondazione Bruno Kessler
Supervisor
s
Luciano Serafini (FBK), Lamberto Ballan
Co-supervisor

Sartori Giuseppe

Project:  Embodied AI with Common-Sense
Full text of the dissertation book can be downloaded from: https://www.research.unipd.it/handle/11577/3475900

Abstract: Embodied AI, or artificial intelligence that is integrated into physical bodies, is an increasingly important area of research due to its potential for creating intelligent agents that can interact with and learn from the real world. These embodied agents can assist humans in a variety of tasks, such as manufacturing, healthcare, and search and rescue operations. For example, industrial robots can be equipped with artificial intelligence to assist in assembly line tasks, allowing for more efficient and accurate production. In healthcare, robots can be used to deliver medication and assist with rehabilitation exercises. In search and rescue operations, robots can be deployed to dangerous or difficult-to-reach areas to gather information and provide assistance. This thesis presents several contributions to the field of Embodied AI, addressing four research questions: (1) how can an agent exploit common-sense knowledge about the environment, (2) how can an agent reuse previously acquired knowledge about a specific environment, (3) how can an agent comply with social rules, and (4) how can an agent acquire knowledge and common-sense rules. To address these research questions, the thesis presents a number of works that provide possible solutions. For example, one work developed a model in which a shared embedding is injected into a Scene-Memory Transformer to improve the ability of an agent to exploit common-sense knowledge about the environment. Another work defined a modular architecture for the Object Goal Navigation task that allows an agent to reuse previously acquired knowledge about a specific environment. Another work presented an agent that is able to navigate cluttered environments while being aware of social rules and the notion of risk. Finally, a preliminary end-to-end framework was presented that can simultaneously learn symbols from perceptions and symbolic functions, which could potentially be applied in an embodied agent to learn how to map perceptions to symbols and common-sense knowledge about an environment. Overall, this thesis makes several important contributions to the field of Embodied AI, providing insights and solutions to a range of challenges faced by intelligent agents operating in the physical world.