# Zeromatter Simulation infrastructure for real-world autonomous systems - Team: Ian Glow (CEO) - Founded: 2021 - Invested: 2024 - Links: [Website](https://zeromatter.com), [LinkedIn](https://www.linkedin.com/in/ianglow/) - Field: Machines that move ## The problem: How do you test a robot on a billion miles it will never drive? ### How a fake world works A robotics simulator lets engineers build the software for a physical robot without the physical machine, which saves cost and time. Programs get written and debugged on a computer, and only the final version goes out onto real hardware. For a drone or a car, that means a practice world that never runs out of runway. At the bottom of that world sits a physics engine, software that approximates how things move and collide. It is an approximation on purpose: real-time engines use simplified calculations and give up some accuracy so they can keep pace. On top sit the sensors. A lidar, for instance, fires a laser at a surface and times how long the light takes to bounce back, so a simulator has to fake that echo, along with the camera image and everything else the robot perceives. Some teams go as far as simulating raindrops on the sensors, dimming light and solar glare. Big systems are rarely one program. In co-simulation, separate pieces of a problem are modelled and run on their own, trading data with each other as the simulation runs. ### Why it is hard **Miles you can't drive.** Fatal crashes are rare compared with the miles people drive, which makes safety slow to prove. To show statistically that a self-driving car is safer than people, it would have to drive hundreds of millions of miles, sometimes hundreds of billions. Even under aggressive assumptions, existing fleets would need tens or hundreds of years to cover that. You cannot simply drive your way to safety. **The tailgater who doesn't brake.** Pick a random tailgating scenario and the tailgater will probably brake in time. The interesting case is the one where they are distracted and don't. Real roads serve up those moments rarely and on their own schedule, so simulation is where developers go looking for them on purpose. **The reality gap.** Real-world data is costly to gather and slow to accumulate, so simulation is attractive: it is a potentially infinite source of data and nobody gets hurt. The catch is that the gap between simulated and real worlds degrades a trained policy once it moves onto a real robot. A robot that aces the practice world can stumble on the first real floor. **Speed against accuracy.** Robots that act on what their sensors tell them are much harder to simulate, because their motion depends on instant readings from the real world. Getting the physics very precise costs more processing power, which is why fast engines cut corners. Yet what happens in simulation has to predict what happens on the road, or the whole exercise is just a very expensive video game. ### What Zeromatter is after Zeromatter wants to speed up how robotics and autonomy get built, tested and trained by leaning on simulation "to the fullest extent possible." Its stated goal is to make serious simulation available to far more teams, as one platform to build, test and train anything. The target is wider than cars. The company lists autonomy, aerospace, automotive, agriculture, drones and green energy among the areas it works in. ### How they go at it **Sensors, faked well.** At the core is a cloud-based sensor simulation platform. The rendering work aims for virtual worlds that look real to the sensors that power autonomous systems, and to human eyes too. A separate sensor characterization effort tries to match each real sensor to its digital twin. **Worlds built by recipe.** The platform generates environments automatically and runs multi-agent co-simulation, so many actors share a scene. Procedural tools, including Houdini, build varied worlds meant to challenge a robot's perception. **Physics for every actor.** An in-house physics engine gives every simulated vehicle, aircraft, drone and robot physically accurate behaviour, with control systems on top. The range runs from cars in city traffic to an airliner taxiing out for take-off. **Checking against hardware.** A robotics team trains policies in the simulator, then validates and iterates on real hardware, with the aim of closing the sim-to-real gap. The company also helps customers certify autonomous systems by documenting how their algorithms were developed, trained and evaluated on its synthetic data. ### Still open Does a simulator need to look real? One well-known experiment trained a real-world object detector, precise enough to guide a robot's grasp, on nothing but simulated images with deliberately unrealistic random textures. The idea is that with enough variety, the real world looks like one more variation. Photorealism and randomness pull in different directions, and both have their champions. When does a simulated mile count as evidence? Test driving alone can't prove safety, so developers need new ways to demonstrate it, and even then it may not be possible to establish safety with certainty. One sign of progress: as developers simulate more variations of a scenario, the spread of outcomes in simulation starts to match what they see on real roads. ### Words used here - **robotics simulator**: Software that stands in for a physical robot and its surroundings so its programs can be written and tested on a computer. - **physics engine**: Software that approximates how objects move, collide and respond to forces. - **lidar**: A sensor that measures distance by timing how long a laser pulse takes to bounce back. - **co-simulation**: Running several separate simulations side by side that exchange data as they go. - **policy**: The learned rule a robot uses to turn what it senses into what it does next. - **sim-to-real gap**: The drop in performance when something trained in simulation meets the real world. - **digital twin**: A software model built to behave like a specific real object, such as one particular sensor. ## About Zeromatter Zeromatter builds the simulation infrastructure that autonomous systems companies use to develop and validate their products before deploying in the real world. The platform provides high-fidelity sensor simulation, automatic environment generation, multi-agent co-simulation, and end-to-end tooling for production simulation workflows. Founded in 2021 in Mountain View by Ian Glow, who spent years leading Autopilot simulation at Tesla before starting Zeromatter to build the same capabilities as a platform. ## Sources 1. [Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?](https://www.rand.org/pubs/research_reports/RR1478.html), RAND Corporation 2. [Simulation City: Introducing Waymo's most advanced simulation system yet for autonomous driving](https://waymo.com/blog/2021/07/simulation-city), Waymo 3. [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://arxiv.org/abs/1703.06907), arXiv 4. [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://arxiv.org/abs/2009.13303), arXiv 5. [Robotics simulator](https://en.wikipedia.org/wiki/Robotics_simulator), Wikipedia 6. [Physics engine](https://en.wikipedia.org/wiki/Physics_engine), Wikipedia 7. [Lidar](https://en.wikipedia.org/wiki/Lidar), Wikipedia 8. [Co-simulation](https://en.wikipedia.org/wiki/Co-simulation), Wikipedia 9. [One platform to build, test and train anything](https://zeromatter.com), Zeromatter 10. [Company](https://zeromatter.com/company), Zeromatter 11. [Join us](https://zeromatter.com/jobs), Zeromatter