The cutting-edge realm of physical AI is unfolding in a warehouse located in San Leandro, California.
This space is home to Encord, a company focused on creating data tools for training artificial intelligence models. Andrew Ceja, who works as a “robotic trainer,” is seen carefully extracting wooden blocks from a precarious tower while wearing a specialized headset that not only captures visuals but also monitors his brain waves during the task.
Encord is among a select group of startups exploring the idea that the primary limitation for humanoid and warehouse robots will not just be their model design, but the lack of real-world physical training data. Instead of simply aiding robotics companies in data management, Encord aims to generate the essential training data that is currently unavailable.
The headset Ceja uses is developed by Zander Labs, a German startup specializing in neuroscience. This device tracks brain activity to reveal mental states such as error detection, intent, and surprise, potentially enhancing the quality of training data for AI models. Currently, Encord’s collaboration with Zander is in the experimental phase; the plan is to compile a unique brain wave-annotated data set, test it against customer robotics models, and assess if it leads to improved performance before considering further development.
Lucas Gehrke, a neuroscientist at Zander, explains that analyzing the brain activity during specific tasks can provide valuable insights for model developers, helping them identify when to implement their most complex robotic solutions.
Vineeth Velmurugan, leading Encord’s robot learning division, describes this initiative as the forefront of efforts to address the data limitations in robotics. With experience from OpenAI and the warehouse automation firm Berkshire Grey, Velmurugan was brought on board to establish the company’s internal data-generation team.
Encord was originally founded to assist businesses focused on machine vision in annotating data and evaluating their AI models. As clients—many of whom are leading robotics firms—adopt a comprehensive learning approach for robotic manipulation tasks, they recognized the necessity to produce their own training data, rather than just manage existing data. Velmurugan noted, “The data simply does not exist.”
The assumption that generative AI could mimic the successes seen in chatbots for robotic applications continually faces similar challenges. Language models have vast amounts of textual data from the internet for training, while finding comparable physical data for teaching neural networks about real-world manipulations remains difficult. Companies developing self-driving technologies may gather this data themselves, but scaling this process is complicated. Video-based training can yield results, yet it often lacks the precision of real-world data. Velmurugan believes that a data set several times larger than YouTube’s video collection is essential to overcome this hurdle, providing context on why generating training data has become a commercial endeavor.
Addressing Your Data Requirements
Robotics companies currently turn to two primary sources for data: “egocentric” videos captured by workers wearing cameras, often supplemented with diverse angles and additional metrics, and information gathered from remotely controlled robots. Encord is actively engaged in both approaches, leveraging egocentric data from various factories worldwide while utilizing their San Leandro facility to explore innovative methods, including brain wave data collection for specialized skill enhancement.
During a recent visit, pilots utilized paired robotic arms—one under human control and the other mimicking its movements—to generate data for tasks like pouring coffee or stacking poker chips. Velmurugan noted that these types of tasks are in high demand among humanoid robotics developers.
The facility is stocked with an array of items, including fake flowers, books, and plastic vegetables, all essential for training robots in household tasks.
At one station, pilot Sofia Infante operates robotic arms to connect and disconnect ethernet cables—routine tasks in data centers that many hope robots can eventually automate. However, I experienced firsthand the limitations of robotic pincers, which lack the versatility and dexterity of human hands.
Another innovative approach being developed by Encord involves sensors attached to the forearm to capture electrical signals from muscles. This method aims to create a 3D representation of hand movements, which typically may not be fully captured in video footage, thereby providing richer data for training.
Encord’s data is meticulously annotated with descriptions of each video’s content—such as “right hand tightens bolt”—to enhance understanding in large language models. Velmurugan estimates that such detailed annotations are worth significantly more compared to less refined data, making the trade-off theoretically advantageous, despite the higher production costs.
However, the financial implications are real; while building AI language models from publicly available text is inexpensive, generating physical training data incurs substantial costs. This reality distinctly separates the economics of physical AI from their linguistic counterparts, as the latter relies heavily on collected rather than manufactured data.
Velmurugan asserts that progress is being made, with Encord’s insights into industry-wide practices allowing it to identify effective strategies and emerging trends for improving physical AI models. The advantage of having a broad view across numerous robotics companies provides Encord with a unique perspective to detect which data strategies are gaining momentum before any individual company can.
With this extensive operation, the dedicated team of pilots at Encord is actively contributing to the foundational development of neural networks. Both Infante and Ceja, having transitioned from another AI data company, bring valuable experience to the team.
Ceja, who previously managed robotic sorters at a waste management facility, now relishes the opportunity to tackle new training challenges for robots, remarking, “It’s something new every day!”
