Markov Robotics
Zero-shot dexterous manipulation
How do you get a robot hand to handle something it has never seen?
Most robot grippers hold things with friction. Plenty of forces act on an object in a robot's grasp, but the main one is friction between the fingers and the thing being held. A common approach is called A grasp that holds an object so that no push or twist can move it without the fingers resisting., which in plain terms means squeezing from enough directions that the object has nowhere to go. The gripping surface can be soft and grippy so it doesn't damage what it holds.
Cameras only get you so far. Weight, stiffness, centre of mass and how slippery something is all require touching it, with some kind of Measuring touch directly, such as pressure and contact, through sensors on the robot's skin or fingertips.. Robots can also feel force through A robot's sense of its own body: where its joints are, how fast they move and what loads they feel., their sense of their own joints.
Handling a case the model never saw an example of in training. is a machine learning term. At test time the model meets things it never saw in training and still has to deal with them. The classic example is a model that has seen horses, has never seen a zebra, but has been told zebras look like striped horses. It gets the zebra. For a hand, the equivalent is a new mug, a new drawer or a new chore, with no fresh round of practice.
Further reading Robot end effector (Wikipedia)Tactile sensor (Wikipedia)Towards Forceful Robotic Foundation Models: a Literature Survey (arXiv)Zero-shot learning (Wikipedia)
- i.
Easy for toddlers
Computers got good at the things we find hard long before the things we find easy. It is comparatively easy to get adult-level performance on intelligence tests or checkers out of a machine, and difficult or impossible to give it the perception and mobility of a one-year-old. The usual explanation is evolution: walking and recognising faces took millions of years to develop, abstract reasoning is recent. Picking up a mug belongs firmly in the old, hard pile.
- ii.
No internet of touch
Language and vision models learned from the web. Robots have no equivalent. Large, broad datasets of robots interacting with the world are hard to come by, and even the biggest collections are a fraction of the size of standard vision and language datasets. So the field has tended to train a separate model for every application, every robot and even every environment, which is a lot of starting over.
- iii.
Simulators that fib a little
The obvious workaround is simulation, since real-world data is slow and costly to gather. But the gap between the simulated and real worlds hurts a policy once it moves onto a real robot. One well-known fix is to randomise physics during training, friction included, so the real world looks like just one more variation. A simulated Shadow Hand trained that way learned to reorient objects and carried the skill over to the physical hand.
- iv.
Touch is slippery
Force and touch are abstract quantities. They can be inferred from many different signals and are often measured and controlled only implicitly. Outside a few tasks such as pouring, peg-in-hole insertion and handling delicate objects, today's imitation-learning models don't yet work at the level where force truly matters.
Further reading Moravec's paradox (Wikipedia)Open X-Embodiment: Robotic Learning Datasets and RT-X Models (arXiv)Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (arXiv)Learning Dexterous In-Hand Manipulation (arXiv)Towards Forceful Robotic Foundation Models: a Literature Survey (arXiv)
Markov's starting point is that "the robotics age is being delayed by staggering data inefficiency", and its bet is that general physical intelligence can be reached at a fraction of the assumed cost.
The target is one mind for any robot and any purpose. It should handle unseen tasks as well as familiar ones, work on any robot body while carrying forward what it has learned, and run fast enough to be useful. That is zero-shot dexterity: a robot you can put in front of new objects and new jobs without collecting demonstrations first.
Further reading Markov Robotics (Markov Robotics)
Markov Robotics has not said enough in public to describe its approach without guessing, so this is left empty rather than filled in.
Does practice on one robot help another?
When 21 institutions pooled data from 22 different robots, a large model trained on the lot showed positive transfer, improving several robots by drawing on experience from other platforms. How far that carries into fine finger work is still being worked out.
How good does a simulator need to be?
Researchers lean on a handful of tricks to close the The drop in performance when a policy trained in a simulator meets real physics., including Varying a simulator's physics and appearance during training so that the real world looks like one more variation., domain adaptation, imitation learning, meta-learning and knowledge distillation. Encouragingly, the simulated hand picked up human-looking habits like finger gaiting with no human demonstrations at all.
Further reading Open X-Embodiment: Robotic Learning Datasets and RT-X Models (abstract) (arXiv)Open X-Embodiment: Robotic Learning Datasets and RT-X Models (arXiv)Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (arXiv)Learning Dexterous In-Hand Manipulation (arXiv)
Markov Robotics is building general-purpose robots capable of dexterous manipulation without task-specific training. Their approach enables robots to handle novel objects and tasks in zero-shot — no demonstrations, no task-specific data collection — making deployment practical across diverse real-world environments.
- force closure
- A grasp that holds an object so that no push or twist can move it without the fingers resisting.
- tactile sensing
- Measuring touch directly, such as pressure and contact, through sensors on the robot's skin or fingertips.
- proprioception
- A robot's sense of its own body: where its joints are, how fast they move and what loads they feel.
- Zero-shot
- Handling a case the model never saw an example of in training.
- sim-to-real gap
- The drop in performance when a policy trained in a simulator meets real physics.
- domain randomization
- Varying a simulator's physics and appearance during training so that the real world looks like one more variation.
- 1Robot end effector · Wikipedia
- 2Tactile sensor · Wikipedia
- 3Towards Forceful Robotic Foundation Models: a Literature Survey · arXiv
- 4Zero-shot learning · Wikipedia
- 5Moravec's paradox · Wikipedia
- 6Open X-Embodiment: Robotic Learning Datasets and RT-X Models · arXiv
- 7Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey · arXiv
- 8Learning Dexterous In-Hand Manipulation · arXiv
- 9Markov Robotics · Markov Robotics
- 10Open X-Embodiment: Robotic Learning Datasets and RT-X Models (abstract) · arXiv