Projects / FOMO-HODOR

Teaching a humanoid to plan chores.
FOMO-HODOR: Foundation Models for Humanoid Domestic Robots

Lead institution
IST-ID
Research unit
ISR Lisboa
Funding
FCT · UT Austin Portugal
Reference
From a spoken sentence to a plan a robot can trust.
Large language models are good at turning a request like “clean the table” into a list of simpler steps. Robots are good at executing steps they can verify. FOMO-HODOR joins the two.
Robot architectures turn sensor data into models that planning software can use, then map each planned task onto what the robot can actually do. The weakness is that every situation has to be encoded in the planner in advance. We use language models to translate natural-language commands into actions, first in plain language, so the robot can tell the person what it will do, and then in a formal language a classical planner can refine and check.
Prior attempts to put language models on robots have been fairly simple. This project builds them into a cohesive robot architecture: a foundation model for humanoid domestic robots, running on our new Booster T1, LOLA.
Two research tasks, one robot.
The project ends with a joint team at RoboCup@Home 2026, in the General-Purpose Service Robot task, where people give the robot spoken commands.
1. Foundation models for robot action planning
Connect natural language to the robot’s internal representations: translate commands into actions, refine and verify them with a classical planner, and keep a working memory of what the robot knows.
2. Language models talking to the rest of the robot
Make interaction robust: combine the robot’s sensors with its memory, adapt speech recognition and synthesis, use vision models inside the language model, and decide what runs onboard or remotely.

LOLA, a humanoid built for homes.
A lightweight Booster T1 with onboard computing for AI, a depth camera for perception and built-in audio for conversation. She lets us try new robot behaviours fast.
Height
1180 mm
Weight
30 kg
Walking speed
0.8 m/s
Degrees of freedom
23, plus two 6-DoF hands
Perception
RealSense D455, 9-axis IMU
Compute
Intel i7-1370P · NVIDIA AGX Orin 32 GB

Qualified for RoboCup@Home 2026.
LisTex United, the joint ISR Lisboa / IST and UT Austin team, is in.
The team.

Pedro Lima
Professor at IST
Principal investigator

Peter Stone
Professor at UT Austin
Co-PI

Bruno Martins
Associate Professor at IST
INESC-ID PI

Justin Hart
Assistant Professor at UT Austin

Afonso Certo
PhD Student at IST

Raghav Arora
PhD Student at UT Austin

Yongwoo Kim
PhD Student at UT Austin

Lingyun Xiao
PhD Student at UT Austin

Rodrigo Coimbra
Research Engineer at ISR Lisboa

Duarte Santos
MSc Student at IST
Partners.

ISR Lisboa, Instituto Superior Técnico
The Intelligent Robots and Systems Group leads the project, bringing its long track record in robot competitions for research and benchmarking.

INESC-ID, Human Language Technologies
Natural-language and speech processing, and a large GPU infrastructure for the language models.

The University of Texas at Austin
Robotics research in the Gates Computer Science building and the Anna Hiss Gym, with a mock apartment for service-robot testing.
Publications.
- Large Language Model-Based Robot Task Planning from Voice Command Transcriptions. Afonso Certo, Bruno Martins, Carlos Azevedo, Pedro Lima. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025.
- LLM+P: Empowering Large Language Models with Optimal Planning Proficiency. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, Peter Stone. arXiv:2304.11477, 2023.
Questions about the project? Write to Pedro Lima, principal investigator.
