You can use different LLMs, either paid like OpenAI or open-source like Qwen/Gemma, and use them to tell the robot things like "pick up the stuffed toy and put it away in the box." Robot will explore, find the necessary objects, and complete the task.

We also have a bunch of tools for learning from demonstration and generally building robotics applications. Tons of docs here: https://github.com/hello-robot/stretch_ai/

On a personal note, I've been a robotics researcher for a very long time, and I think, unquestionably, that the biggest problem with robotics research is that we don't have good software that makes it easy to share and test models, and we don't have common platforms. This is sort of my attempt to change that.

Twitter thread with a bunch of videos: https://twitter.com/chris_j_paxton/status/186399720688316423...