The Robot Puppeteers Of Silicon Valley Teaching Humanoids How To Make Your Morning Coffee

Fernando Flores can spend eight hours a day pouring the same cup of coffee.

He is not a barista. He’s a robot puppeteer, trying to train humanoids.

He manipulates mechanical arms remotely, using hand and arm sensors to make them pick up a pot of coffee, pour it into a mug and put the pot back in the coffee maker. Flores checks for spills, then empties the mug back into the pot by hand and does it again — hundreds of times.

“The repetitiveness, it can cause some discomfort,” said Flores, who has the title of senior robotic pilot at San Francisco startup Encord. “It becomes second nature after a while.”

This Sisyphus of Silicon Valley is on the front lines of a rapidly expanding industry of robot trainers, preparing to teach and operate the army of humanoid robots scheduled to march out of nearby factories in the coming year. Encord practices, records and sells data about movement to the companies racing to bring humanoids to homes, offices and factories.

If tech companies’ optimistic plans are to be believed, a swarm of American-built robots is about to hit the market.

Tesla’s Fremont factory stopped car production this year to make way for production lines for its Optimus robots, with unbelievable plans to ramp up capacity to 1 million units a year. Palo Alto-based 1X Technologies is already manufacturing its 66-pound, 5-foot-6 humanoid named Neo at its factory in Hayward. The company received 10,000 preorders, and its first shipment is expected later this year. Figure AI’s humanoid factory in San Jose has increased its manufacturing capacity to produce one Figure 03 robot an hour, with the goal of producing 12,000 a year.

Fernando Flores demonstrates the articulation of a robot performing a whisking motion at Encord on May 21.

(Paul Kuroda / For The Times)

Goldman Sachs projects the global market for humanoids could reach $38 billion by 2035.

The AI of these humanoid robots needs an immense amount of data on human movement. How humans write, speak, code and compose was easily scraped off the internet, but the bots need more information to master how to stand, step, lift, squeeze, pour and perform other physical movements. That is where companies like Encord come in.

The $10 billion invested in robotics in 2026, according to CB Insights, has spawned an industry focused on training robots. Initially, that meant humans strapping iPhones to their foreheads, recording actions like cooking, cleaning and performing household chores. That, however, doesn’t capture the exact torque, force and grip required for a robot hand to work flawlessly.

Now, humans are directly guiding robots through expensive rigs that let them control the robots’ movements. Data collected using robot arms offer richer insights into motor skills and object manipulation. Encord charges clients up to $1,000 per hour for training data.

The information gathered from trainers controlling robots is “super important to bridge the next level of learning,” where robots will learn to correct mistakes and do the chores on their own, said Vineeth Velmurugan, head of robotics learning at Encord.

The company is already working with some of the top companies in robotics, but said it couldn’t share most names. Among the clients it could mention were Toyota Research Institute and Weave, which already has laundry-folding robots in a few homes.

Brian Gonzalez pulls an ethernet cable using a robotic arm at Encord

Many of the new robotic data companies are focusing on industrial use cases. Robots can perform better in a structured, predictable environment, like a factory or warehouse.

Home tasks are tougher, as layouts and tasks are more varied and messy. While many bots have mastered walking, they still struggle to open doors, fridges and washing machines smoothly. They don’t know where or how to grasp a doorknob, handle or door edge or how much pulling, pushing or twisting force to apply.

Flores has mastered making the robot arms pour coffee, but he still often spills. When that happens, he deletes records of the attempt.

“Typically, we don’t want any mistakes,” he said. “If we have more than three consecutive mistakes within a 15‑second window, that’s not going to be good data.”

Inside Encord’s test facility in Hayward, it has replicated a standard American home with a fully furnished living room, kitchen and bathroom.

In the living room, a pilot rearranges an untidy study desk. She first scatters AA-size batteries, pens and scissors on the table, and walks back to the nearby control rig to make the robot arms place each one inside the tray of a desk organizer.

Depending on the day’s training, the pilots could be opening and closing refrigerator doors, whisking liquids in a bowl, sorting silverware or turning a water faucet on and off over and over until the robot arms get it right.

Cortney Weintz, left, and Tony Schiller record data with cameras at Encord.

In another corner of the facility, people wearing smart glasses place and pick up playing cards and sort plastic plates by hand, collecting first-person videos.

One key skill for the coming bot invasion: plugging in cables.

Companies want robots that can crawl into duct spaces, identify ports and plug cables to help build the massive data centers needed for AI. Encord replicated a real data center server rack, where an operator inserts blue cables into penny-sized sockets all day.

Many companies have entered this business. Meta-backed Scale AI and Palo Alto-based Micro1 are major players in the space. China has more than 40 state-owned robot data-collection facilities where hundreds of on-site humans mimic train bots how to move in the real world.

In Watertown, Mass., Tutor Intelligence has set up a 100-robot facility dedicated to harvesting movement data. Its robot arms, which are being trained to do factory work, are controlled by a human team split across Mexico, the Philippines and Boston. This is in part to train its robot, Sonny, which will hit the market later this year.

Elaine Batchlor sorts screws and bolts with a robot in a mockup at Encord.

“We built the Data Factory to bootstrap the initial intelligence for the Sonny robot, so that we can begin to deploy Sonny into the field,” said Josh Gruenstein, co-founder of Tutor. Ten of its remote operators are based in Boston, and the rest are international.

Remote operation is emerging as an integral part of the humanoid robot business. Employing teleoperators in countries where wages are much lower than in the U.S. could, in theory, mean a robot controlled by a human in another country could do a task at a fraction of the cost of having an American do it.

This month, a humanoid robot cleaning service in San Francisco called Gatsby completed a robot cleaning of a U.S. home using a teleoperator in Mexico.

The technology is still evolving, said Aron Frishberg, co-founder of Gatsby, but being a first mover means Gatsby is getting more training.

“There’s obviously stuff that goes wrong,” he said. “It’s really hard to get precise hand movements or arm movements and grab something.”

Encord co-founder Ulrik Hansen said it will be setting up a teleoperations center in its Hayward facility in the next three months. Even as more robots are deployed and master increasingly sophisticated tasks, they will still need humans to occasionally take control remotely.

“They will need some exception handling when they get things wrong,” he said.

Hundreds of teleoperators will learn where the system succeeds, where it breaks and step in when needed. Once those patterns emerge, Hansen said, they can move teleoperations to cheaper locations abroad or to the Midwest.

Back in Hayward, Flores created new coffee-pouring challenges for his robot arms. He changed what was on the counter around the coffee maker and moved the mug to different spots. It takes a lot of know-how to puppet and train a robot, he said.

“A lot of people would (guess) this might be easy, this is dumb,” Flores said. “There actually is thought here. There actually is critical thinking.”