Our goal is to catalyze tangible and substantial improvements in robot capabilities, especially in human–robot scenarios, with research at the confluence of learning, modeling, and robotics. We accomplish this from two perspectives:
- In the first, we leverage recent advances in learning and modeling to develop sophisticated, accessible, and generalizable robot technologies. Our focus here is primarily directed toward robots that operate in human environments. The ability of robots to interact or even cooperate with humans are considerations that are often neglected in the learning and modeling communities.
- Complementarily, we draw upon our expertise in robotics and human–robot interaction to inform foundational research in artificial intelligence—particularly in natural language processing and reinforcement learning. Our experience with real-world robotic systems allows us to underpin our research with strong empirical motivations, an approach that is conspicuously scant in much of the existing literature.
Contents
1. Projects
1.1. Non-Prehensile Manipulation
Students: Anuj Pasricha, Yi-Shiuan Tung
Publications:
- A. Pasricha, Y. Tung, B. Hayes, and A. Roncone, “PokeRRT: Poking as a skill and failure recovery tactic for planar non-prehensile manipulation” in Robotics and Automation Letters and 2022 IEEE International Conference on Robotics and Automation (ICRA), 2022. [PDF] [BIB]
Non-prehensile manipulation (i.e., manipulation that does not involve grasping) can significantly expand the operational space of a robot. We posit that robots need to leverage non-prehensile manipulation as part of their skill set if they are to achieve human-level dexterity. Our past work introduced a novel planner that uses poking as a skill and failure recovery tactic synergistically with grasping. Moving forward, we are building hybrid models for manipulation in which physics-based and learning-based approaches complement each other, toward generating a repository of skills that robots can then use to engage in more complex, affordance-informed task planning and manipulation.
1.2. Natural Language Grounding and Skill Transfer
Students: Stéphane Aroca-Ouellette
Publications:
- S. Aroca-Ouellette, C. Paik, A. Roncone, and K. Kann, “PROST: Physical Reasoning of Objects through Space and Time”, 2021. In Findings of the Association for Computational Linguistics: ACL-IJCNLP2021, [PDF] [BIB]
Natural language is the easiest and most generalizable way for humans to specify a task, provide new information, and convey intentions. Being able to leverage language for task specification and skill transfer would greatly increase the abilities of current robots. Concurrently, current language models fail to understand language as humans do, which we hypothesize is caused a lack of real-world experience. To this end, we aim at bridging the gap between the field of robotics and NLP to produce robots that can act and learn through language, and who in turn will generate experiences for it develop a richer understanding of language.
1.3. Multi-Agent Reinforcement Learning
Students: Guohui Ding, Joewie J. Koh
Publications:
- J. J. Koh, G. Ding, C. Heckman, L. Chen, A. Roncone, “Cooperative control of mobile robots with Stackelberg learning,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020. [PDF] [BIB]
- G. Ding, J. J. Koh, K. Merckaert, B. Vanderborght, M. M. Nicotra, C. Heckman, A. Roncone, L. Chen, “Distributed reinforcement learning for cooperative multi-robot object manipulation,” in 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020. [PDF] [BIB]
Reinforcement learning (RL), which allows an agent to learn from interactions with its environment, presents an increasingly promising alternative to traditional model-based control. However, much of the work applying RL to robotics has been in the single-agent paradigm—despite the pervasiveness of multi-agent robotic systems in the real world. We are most interested in developing algorithms for controlling heterogeneous teams that cooperate to accomplish common goals, with the aim of enabling multi-robot systems to be imbued with capabilities that are more than the sum of their parts. Moreover, we envision multi-agent systems serving as the setting for the next wave of progress in RL research, and are further driven by the belief that multi-agent RL can provide a theoretical basis for understanding the application of RL in human–robot contexts.
1.4. Learning Discourse Policies for Dialog Management
Students: Joewie J. Koh, Kaleb Bishop
The NSF National AI Institute for Student-AI Teaming (iSAT) is a multi-site interdisciplinary institute with the vision of developing artificial intelligence as a social, collaborative partner that helps both students and teachers make learning more effective, engaging, and equitable. As a part of this institute, we conduct foundational research on dialog management for AI-based conversation participation and facilitation. Specifically, we are studying how reinforcement learning might be applied to autonomously generate and fine-tune discourse policies.
2. Publications
2023 International Conference on Autonomous Agents and Multiagent Systems [AAMAS]
London, Uk, May 28-June 02
2022 Workshop on Natural Language Processing for Conversational AI @ ACL 2022
Dublin, Ireland, May 27
2022 IEEE Robotics Automation and Letters [RA-L] + ICRA
Philadelphia, PA, U.S.A., May 23-27
2021 IROS 2021 Workshop on Impact-Aware Robotics
Prague, Czech Republic, October 1
2021 Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Online and Punta Cana, Dominican Republic, November 2021
2021 Findings of the Association for Computational Linguistics
Online, August 1-6
2020 IEEE/RSJ International Conference on Intelligent Robots and Systems [IROS]
Las Vegas, NV, U.S.A., October 25-29
2020 International Conference on Autonomous Agents and Multiagent Systems [AAMAS]
Auckland, New Zealand, May 9-13