Intelligent Robots and Systems Group · ISR Lisboa · Instituto Superior Técnico

Projects / FOMO-HODOR

FOMO-HODOR project logo

Teaching a humanoid to plan chores.

FOMO-HODOR: Foundation Models for Humanoid Domestic Robots

LOLA, the Booster T1 humanoid used in FOMO-HODOR, in a kitchen
LOLA, ready for chores

Lead institution

IST-ID

Research unit

ISR Lisboa

Funding

FCT · UT Austin Portugal

Reference

2024.14126.UTA

From a spoken sentence to a plan a robot can trust.

Large language models are good at turning a request like “clean the table” into a list of simpler steps. Robots are good at executing steps they can verify. FOMO-HODOR joins the two.

Robot architectures turn sensor data into models that planning software can use, then map each planned task onto what the robot can actually do. The weakness is that every situation has to be encoded in the planner in advance. We use language models to translate natural-language commands into actions, first in plain language, so the robot can tell the person what it will do, and then in a formal language a classical planner can refine and check.

Prior attempts to put language models on robots have been fairly simple. This project builds them into a cohesive robot architecture: a foundation model for humanoid domestic robots, running on our new Booster T1, LOLA.

Two research tasks, one robot.

The project ends with a joint team at RoboCup@Home 2026, in the General-Purpose Service Robot task, where people give the robot spoken commands.

1. Foundation models for robot action planning

Connect natural language to the robot’s internal representations: translate commands into actions, refine and verify them with a classical planner, and keep a working memory of what the robot knows.

2. Language models talking to the rest of the robot

Make interaction robust: combine the robot’s sensors with its memory, adapt speech recognition and synthesis, use vision models inside the language model, and decide what runs onboard or remotely.

LOLA, the Booster T1 humanoid, standing in the lab
Hi, I’m LOLA.

LOLA, a humanoid built for homes.

A lightweight Booster T1 with onboard computing for AI, a depth camera for perception and built-in audio for conversation. She lets us try new robot behaviours fast.

Height

1180 mm

Weight

30 kg

Walking speed

0.8 m/s

Degrees of freedom

23, plus two 6-DoF hands

Perception

RealSense D455, 9-axis IMU

Compute

Intel i7-1370P · NVIDIA AGX Orin 32 GB

LisTex United team logo

Qualified for RoboCup@Home 2026.

LisTex United, the joint ISR Lisboa / IST and UT Austin team, is in.

The team.

Pedro Lima

Pedro Lima

Professor at IST

Principal investigator

Peter Stone

Peter Stone

Professor at UT Austin

Co-PI

Bruno Martins

Bruno Martins

Associate Professor at IST

INESC-ID PI

Justin Hart

Justin Hart

Assistant Professor at UT Austin

Afonso Certo

Afonso Certo

PhD Student at IST

Raghav Arora

Raghav Arora

PhD Student at UT Austin

Yongwoo Kim

Yongwoo Kim

PhD Student at UT Austin

Lingyun Xiao

Lingyun Xiao

PhD Student at UT Austin

Rodrigo Coimbra

Rodrigo Coimbra

Research Engineer at ISR Lisboa

Duarte Santos

Duarte Santos

MSc Student at IST

Partners.

ISR Lisboa

ISR Lisboa, Instituto Superior Técnico

The Intelligent Robots and Systems Group leads the project, bringing its long track record in robot competitions for research and benchmarking.

INESC-ID

INESC-ID, Human Language Technologies

Natural-language and speech processing, and a large GPU infrastructure for the language models.

The University of Texas at Austin

The University of Texas at Austin

Robotics research in the Gates Computer Science building and the Anna Hiss Gym, with a mock apartment for service-robot testing.

Publications.

  1. Large Language Model-Based Robot Task Planning from Voice Command Transcriptions. Afonso Certo, Bruno Martins, Carlos Azevedo, Pedro Lima. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025.
  2. LLM+P: Empowering Large Language Models with Optimal Planning Proficiency. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, Peter Stone. arXiv:2304.11477, 2023.

Questions about the project? Write to Pedro Lima, principal investigator.