We are probably 5-6 orders of magnitude short of the real robotic data we will need to train a foundational model for robotics, so how do we get that? I believe simulation or video can be a complement, but there is no substitution for a ton of real robotic data.
I’ve been exploring approaches to scale robotic teleoperation, traditionally relegated to slow high-value use cases (nuclear decommissioning, healthcare). Here’s a short video from a raw testing session (requires a lot of explanation!):
What is happening here?
First of all, this is true robotic teleoperation (often people confuse controlling a robot in line-of-sight with teleoperation): I am controlling a robotic arm via a VR teleoperation setup without wearing it, to improve ergonomics, but watching at camera feeds. Over wifi, with a simulated 300ms latency + 10ms jitter (international round trip latency, say UK to Australia).
On the right a pure teleoperation run is shown. Disregard the weird “dragging” movements, they are a drag-and-drop implementation I built to allow the operator to reposition the human arm in a more favorable position without moving the robotic arm. Some of the core issues with affordable remote teleoperation are reduced spatial 3D awareness, human-robot embodiment gap, and poor force-tactile feedback. Combined with network latency and limited robotic hardware dexterity they result in slow and mentally draining operations. Often teleoperators employ a “wait and see” strategy similar to the video, to reduce the effects of latency and reduced 3D awareness. It’s impractical to teleoperate a robot for hour-long sessions.
On the left an AI helps the operator twice to sustain long sessions at a higher pace. There is an "action AI" executing individual actions such as picking (the “action AI” right now is a mixture of VLAs [Vision Language Action models], computer vision, motion planning, dynamic motion primitives; in the future it will be only VLAs) and a "human-in-the-loop AI", which is dynamically arbitrating when to give control to the teleoperator or to the action AI. The final movement is the fusion of the AI and the operator movement, with some dynamic weighting based on environmental and contextual factors. In this way the operator is always in control and can handle all the edge cases that the AI is not able to, while the AI does the lion share of the work in subtasks where enough data is already available.
Currently it can speed up experienced teleoperators by 100-150% and much more for inexperienced teleoperators. The reduction in mental workload is noticeable from the first few sessions. An important challenge is speeding up further vs a human over long sessions. Technically, besides AI, it’s about improving robotic hardware, 3D telepresence, network optimisation, teleoperation design and ergonomics.
I see this effort as part of a larger vision to improve teleoperation infra, scale up robotic data collection and deploy general purpose robots everywhere.
About me, I am currently head of AI in Createc, a UK applied robotic R&D lab, in which I built hybrid AI systems. Also 2x startup founder (last one was an AI-robotics exit).
I posted this to gather feedback early. I am keen to connect if you find this exciting or useful! I am also open to early stage partnerships.