Are brain waves the next solution to physical AI?

The Frontiers of Physical AI It’s a game of Jenga in a warehouse in San Leandro, California.

This warehouse is already occupied harmonya company that builds data tools used to train artificial intelligence models. Andrew Seaga is a pilot — the company’s term for its robotic trainers — carefully pulling wooden blocks from a tottering tower while wearing a headset with a camera that tracks what he sees. This alone is fairly common in collecting robot training data, but this headset includes sensors that measure its brain waves as it carefully disassembles the mass tower.

Incord is one of a small but growing number of startups betting that the next real limitation for humanoid and warehouse robots will not be model engineering but instead the sheer scarcity of real-world physical training data. Instead of just helping robotics companies manage the data they have, Encord is building a company around manufacturing the data they don’t have.

The brainwave headset that Ceja wears was designed by Zander Labsa German neuroscience startup that is betting that measuring brain activity — to infer mental states like error, intention, and surprise — can create a more useful data set for training models. Encord’s work with Zander is currently in trial run; The goal, Incord says, is to build an initial brainwave-labeled dataset, run it through customer robotics models, and evaluate whether it actually improves performance before deciding whether to scale it up.

The amount of brain activity used at any point during a given task provides clues to model builders trying to figure out when they need to deploy their models with peak effort, says Lukas Gehrke, the Zander neuroscientist supervising the work.

This is the “bleeding edge” of efforts to solve the robotics data bottleneck, according to Vineeth Velmurugan, head of robotics learning at Encord. Velmurugan, a veteran of OpenAI’s robotics lab and Berkshire Gray, the warehouse automation company, joined Encord to build out the company’s internal data creation team.

Encord was founded to help companies building machine vision applications explain data and evaluate models. When their clients — whom Velmurugan says work with several leading robotics companies, but he is not authorized to name them — began applying end-to-end learning to robot manipulation tasks, executives realized they would have to produce the training data themselves, rather than just manage it. “The data simply doesn’t exist,” Velmurugan said.

Betting that generative AI can do for bots what it does for chatbots keeps hitting the same wall. LLMs are built on the entire Internet text, and more. Finding the same raw material to teach neural networks about physical manipulation is difficult as self-driving car companies collect themselves, but it’s even harder to scale. Video training can work, but it lacks the accuracy of real-world data. Velmurugan says it would take a data set five times the size of a YouTube video collection to hack it, a scale that helps explain why data generation itself has become a business, not just a research problem.

Feed your selfish data needs

Companies that make robot brains are now turning to two main sources: “egocentric” video collected by workers wearing cameras, often enhanced with additional camera angles and other metrics. Data collection from robots Operate remotely. Encord does both, pulling in subjective data from many factories around the world, and using its facility in San Leandro to experiment with new modalities, such as brain waves, or collect datasets on specific skills for fine-tuning.

When TechCrunch visited, pilots were using leader-follower devices — paired robotic arms, one controlled directly by a human operator and the other mimicking its movements — to generate data on tasks like pouring coffee from a pot into cups (extremely boring) and stacking poker chips. “Every robotics company has asked us for these parts,” says Velmurugan.

Storage shelves held boxes of fake flowers in vases, books, plastic vegetables, trays and scoops of kitty litter, and bags and bundles of wire, stock traded to train jugglers for household tasks.

At one of these stations, another pilot, Sofia Infante, maneuvers robotic arms to connect and disconnect Ethernet cables from the back of a server — the kind of work that data center operators would like to see automated, if only robots could handle it with the precision required. Looking at the controls, I could see why this was so elusive: pincers are much less dexterous than human fingers and lack the degrees of freedom we take for granted in our arms.

Another new data method being developed by Encord uses an array of sensors strapped to the forearm to detect electrical signals in muscles. Video taken of human hands manipulating objects typically doesn’t capture the entire hand, but Velmurugan hopes to build a 3D depiction of where the hand is at any given time based on the arm’s sensors, creating a more robust understanding of the models.

Encord datasets are annotated with physical descriptions of what each video contains—“right hand tightens the bolt”—to help LLM-based models understand what’s happening. Velmurugan estimates that this kind of dense annotation is worth 100 times as much “junk ego data” to train specific tasks, and costs only 20 times as much to produce, which is a good business on paper.

But “20 times more” is still real money, and that’s the problem: deleting text from the Internet, the way LLM makers built their models by pulling from Stack Overflow and the rest of the web, costs Frontier Labs almost nothing. Physical training data is not generated, this is the maximum comparison between physical AI and MBA. This type of data needs to be synthesized, not just collected, and this changes the economics of building these models.

Velmurugan says progress is being made, and through Encord’s vision of software across the industry, he’s been able to see startups and leading labs alike discover what works and what doesn’t to improve physical AI models. This vantage point – located between several robotics companies at once – is also part of Encord’s offering. It can discover data technologies that are gaining industry-wide traction before any single customer can.

That will keep the dozen or so pilots at the Encord facility busy. Both Infante and Sega are part of a thriving workforce developing the building blocks of neural networks. They previously worked at Scale, another AI data annotation company, before joining Encord.

Sega worked for a waste management company where his interest in technology made him responsible for keeping the automated garbage sorting machine in good working order. Now, with the Jenga Tower collapsing, he says he enjoys the challenge of solving the robots’ training tasks — “It’s something new every day!”