- Shape Estimation: 1.92 mm error (41% improvement over best baseline)
- Trajectory Tracking: 3.29 mm error (47% improvement over strain-tracking)
- Task Performance: Average 82% success rate across all tasks
KineSoft combines shape-based imitation, proprioceptive sensing, and a shape-conditioned controller to achieve dexterous in-hand manipulation with compliant soft robot hands.
Underactuated soft robot hands offer inherent safety and adaptability advantages over rigid systems. While imitation learning shows promise for acquiring complex dexterous manipulation skills, adapting existing methods to soft robots presents unique challenges in state representation and data collection. We propose KineSoft, a framework for direct kinesthetic teaching of soft robotic hands that leverages their natural compliance as a skill-teaching advantage rather than only as a control challenge. KineSoft makes three key contributions: (1) a shape-based imitation learning framework that uses proprioceptive feedback to ground diffusion-based policies, (2) a low-level shape-conditioned controller that enables precise tracking of desired shape trajectories, and (3) a sim-to-real learning approach for soft-robot mesh shape sensing with an internal strain-sensing array. In physical experiments, KineSoft significantly outperforms strain-signal baselines across six in-hand manipulation tasks involving both rigid and deformable objects.
KineSoft grounds diffusion-based policies in rich proprioceptive mesh representations, enabling efficient learning from kinesthetic demonstrations.
Our controller translates vertex-level shape errors into tendon commands, bridging the demonstration-to-execution gap typical in soft robots.
A self-supervised domain-alignment procedure achieves <2 mm median shape error when transferring a simulation-trained estimator to real hardware.
Soft robots' natural compliance enables intuitive kinesthetic teaching, where human demonstrators can directly manipulate the hand into desired configurations. KineSoft leverages this advantage for skill acquisition.
Our proprioceptive model uses internal strain sensors embedded in soft fingers to estimate real-time mesh deformations. A FoldingNet-based architecture predicts per-vertex displacements from sensor readings.
We use CMA-ES optimization to align simulated sensor models with real-world measurements, minimizing the Chamfer distance between observed and predicted point clouds. This enables robust transfer of simulation-trained models to physical hardware.
The controller tracks desired shape trajectories by projecting vertex-level shape errors onto tendon actuation directions, enabling precise deformation control at 100 Hz.
We evaluated KineSoft on six challenging manipulation tasks:
Success Rate: 85%
(Baseline: 0%)
Success Rate: 70%
(Baseline: 15%)
Success Rate: 80%
(Baseline: 35%)
Success Rate: 100%
(Baseline: 90%)
Success Rate: 95%
(Baseline: 65%)
Success Rate: 65%
(Baseline: 5%)
We use the Multifinger Omnidirectional End-effector (MOE) platform with embedded conductive rubber sensors. Each finger contains 4 strain sensors providing real-time proprioceptive feedback at 400 Hz.
Training data generated using SOFA framework with Neo-Hookean hyperelastic material models. Random external forces simulate contact-rich interactions for robust shape estimation.
KineSoft demonstrates that soft robots' inherent compliance can be leveraged as an advantage for skill learning through kinesthetic teaching. Our shape-based hierarchical approach successfully bridges the gap between demonstration and execution modes, enabling effective dexterous manipulation.
Key Insights:
This work opens new possibilities for intuitive programming of soft robotic systems in applications requiring safe, compliant interaction with delicate objects and humans.
@inproceedings{yoo2025kinesoft,
title={KineSoft: Learning Proprioceptive Manipulation Policies with Soft Robot Hands},
author={Yoo, Uksang and Francis, Jonathan and Oh, Jean and Ichnowski, Jeffrey},
booktitle={Proceedings of the 9th Conference on Robot Learning (CoRL)},
year={2025}
}We thank the reviewers for their valuable feedback and the CMU Robotics Institute for supporting this research.