Learning, Modeling, and Robotics

Bridging the gap between robotics and artificial intelligence

Our goal is to catalyze tangible and substantial improvements in robot capabilities, especially in human–robot scenarios, with research at the confluence of learning, modeling, and robotics. We accomplish this from two perspectives:

  • In the first, we leverage recent advances in learning and modeling to develop sophisticated, accessible, and generalizable robot technologies. Our focus here is primarily directed toward robots that operate in human environments. The ability of robots to interact or even cooperate with humans are considerations that are often neglected in the learning and modeling communities.
  • Complementarily, we draw upon our expertise in robotics and human–robot interaction to inform foundational research in artificial intelligence—particularly in natural language processing and reinforcement learning. Our experience with real-world robotic systems allows us to underpin our research with strong empirical motivations, an approach that is conspicuously scant in much of the existing literature.


1. Projects

1.1. Non-Prehensile Manipulation

Students: Anuj Pasricha, Yi-Shiuan Tung


  • A. Pasricha, Y. Tung, B. Hayes, and A. Roncone, “PokeRRT: Poking as a skill and failure recovery tactic for planar non-prehensile manipulation” in Robotics and Automation Letters and 2022 IEEE International Conference on Robotics and Automation (ICRA), 2022. [PDF] [BIB]

Non-prehensile manipulation (i.e., manipulation that does not involve grasping) can significantly expand the operational space of a robot. We posit that robots need to leverage non-prehensile manipulation as part of their skill set if they are to achieve human-level dexterity. Our past work introduced a novel planner that uses poking as a skill and failure recovery tactic synergistically with grasping. Moving forward, we are building hybrid models for manipulation in which physics-based and learning-based approaches complement each other, toward generating a repository of skills that robots can then use to engage in more complex, affordance-informed task planning and manipulation.

1.2. Natural Language Grounding and Skill Transfer

Students: Stéphane Aroca-Ouellette


  • S. Aroca-Ouellette, C. Paik, A. Roncone, and K. Kann, “PROST: Physical Reasoning of Objects through Space and Time”, 2021. In Findings of the Association for Computational Linguistics: ACL-IJCNLP2021, [PDF] [BIB]

Natural language is the easiest and most generalizable way for humans to specify a task, provide new information, and convey intentions. Being able to leverage language for task specification and skill transfer would greatly increase the abilities of current robots. Concurrently, current language models fail to understand language as humans do, which we hypothesize is caused a lack of real-world experience. To this end, we aim at bridging the gap between the field of robotics and NLP to produce robots that can act and learn through language, and who in turn will generate experiences for it develop a richer understanding of language.

1.3. Multi-Agent Reinforcement Learning

Students: Guohui Ding, Joewie J. Koh


  • J. J. Koh, G. Ding, C. Heckman, L. Chen, A. Roncone, “Cooperative control of mobile robots with Stackelberg learning,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020. [PDF] [BIB]
  • G. Ding, J. J. Koh, K. Merckaert, B. Vanderborght, M. M. Nicotra, C. Heckman, A. Roncone, L. Chen, “Distributed reinforcement learning for cooperative multi-robot object manipulation,” in 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020. [PDF] [BIB]

Reinforcement learning (RL), which allows an agent to learn from interactions with its environment, presents an increasingly promising alternative to traditional model-based control. However, much of the work applying RL to robotics has been in the single-agent paradigm—despite the pervasiveness of multi-agent robotic systems in the real world. We are most interested in developing algorithms for controlling heterogeneous teams that cooperate to accomplish common goals, with the aim of enabling multi-robot systems to be imbued with capabilities that are more than the sum of their parts. Moreover, we envision multi-agent systems serving as the setting for the next wave of progress in RL research, and are further driven by the belief that multi-agent RL can provide a theoretical basis for understanding the application of RL in human–robot contexts.


Learning to cooperate with asymmetric perceptual capabilities.

1.4. Learning Discourse Policies for Dialog Management

Students: Joewie J. Koh, Kaleb Bishop

The NSF National AI Institute for Student-AI Teaming (iSAT) is a multi-site interdisciplinary institute with the vision of developing artificial intelligence as a social, collaborative partner that helps both students and teachers make learning more effective, engaging, and equitable. As a part of this institute, we conduct foundational research on dialog management for AI-based conversation participation and facilitation. Specifically, we are studying how reinforcement learning might be applied to autonomously generate and fine-tune discourse policies.


iSAT's vision for student-AI teaming and classroom orchestration.

2. Publications

2022 Workshop on Natural Language Processing for Conversational AI @ ACL 2022

Dublin, Ireland, May 27

Open-Domain Dialogue Generation: What We Can Do, Cannot Do, And Should Do Next [PDF] [BIB]

Katharina Kann, Abteen Ebrahimi, Joewie J. Koh, Shiran Dudy, Alessandro Roncone

2022 IEEE Robotics Automation and Letters [RA-L] + ICRA

Philadelphia, PA, U.S.A., May 23-27

PokeRRT: Poking as a Skill and Failure Recovery Tactic for Planar Non-Prehensile Manipulation [PDF] [BIB]

Anuj Pasricha, Yi-Shiuan Tung, Bradley Hayes, Alessandro Roncone

2021 IROS 2021 Workshop on Impact-Aware Robotics

Prague, Czech Republic, October 1

PokeRRT: A Kinodynamic Planning Approach for Poking Manipulation [PDF] [BIB]

Anuj Pasricha, Yi-Shiuan Tung, Bradley Hayes, Alessandro Roncone

2021 Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

Online and Punta Cana, Dominican Republic, November 2021

The World of an Octopus: How Reporting Bias Influences a Language Model’s Perception of Color [PDF] [BIB]

Cory Paik, Stéphane Aroca-Ouellette, Alessandro Roncone, Katharina Kann

2021 Findings of the Association for Computational Linguistics

Online, August 1-6

PROST: Physical Reasoning of Objects through Space and Time [PDF] [BIB]

Stéphane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann

2020 IEEE/RSJ International Conference on Intelligent Robots and Systems [IROS]

Las Vegas, NV, U.S.A., October 25-29

Cooperative Control of Mobile Robots with Stackelberg Learning [PDF] [BIB]

Joewie J. Koh*, Guohui Ding*, Christoffer Heckman, Lijun Chen, Alessandro Roncone

2020 International Conference on Autonomous Agents and Multiagent Systems [AAMAS]

Auckland, New Zealand, May 9-13

Distributed Reinforcement Learning for Cooperative Multi-Robot Object Manipulation [PDF] [BIB]

Guohui Ding*, Joewie J. Koh*, Kelly Merckaert, Bram Vanderborght, Marco M. Nicotra, Christoffer Heckman, Alessandro Roncone, Lijun Chen
Extended abstract