Capturing Motion,
Crafting Stories

Figure 1. FreeTacMan handheld gripper and contact-rich manipulation scenarios.

Robots that grasp fragile objects, insert plugs, write, stamp, or classify textures need more than visual recognition. They must also understand contact, gripper motion, pose changes, and the timing relationship between touch and action. Recently, the research team led by Prof. Hongyang Li at the University of Hong Kong and Shanghai Qi Zhi Institute published a paper entitled “FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation” at the 2026 IEEE International Conference on Robotics and Automation (ICRA 2026), provides a useful reference for this challenge.

FreeTacMan is described by the project team as a robot-free, human-centric visuo-tactile data collection system. It enables human operators to collect demonstrations with a wearable handheld gripper, while visual input, tactile feedback, motion control, and pose information are converted into data that can support robot imitation learning. In this research, NOKOV motion capture was used to obtain high-precision 6D pose data from the collection interface, making it a key measurement layer for building synchronized multimodal trajectories.

Why does contact-rich manipulation need more than vision?

Contact-rich manipulation covers robot tasks where interaction with an object or surface is continuous and sensitive. A robot may need to insert a USB plug, place a fragile cup without damage, press a stamp clearly, or follow a calligraphy trajectory. In these tasks, the robot cannot rely only on what a camera sees. It also needs to know when contact happens, how the contact changes, and how the gripper moves in space.

This is why visual-tactile data is important. Vision records the environment and object state. Tactile feedback records local contact, deformation, slip, and texture cues. Pose tracking describes how the gripper translates and rotates during the demonstration. When these data streams are synchronized, they create a more complete training signal for robot policy learning.

What problem does FreeTacMan solve?

Traditional robot training data collection often depends on robot-mounted sensors, teleoperation rigs, AR/VR systems, or handheld devices based on SLAM and IMU localization. These methods can work, but the FreeTacMan project highlights several recurring limitations: inefficient data collection, limited sensor setups, complex calibration, indirect tactile feedback, and motion tracking drift.

FreeTacMan changes the workflow by moving data collection away from the robot body. A human operator wears or holds a gripper with modular visuo-tactile sensors and performs the contact-rich task directly. This preserves natural human control and real-time tactile feedback while still producing data that can be transferred to robot execution.

What makes the FreeTacMan system useful for robot learning?

The project page highlights three main parts of the system: a modular visuo-tactile hardware sensor, an in-situ robot-free real-time tactile data-collection system, and visuo-tactile policy learning with tactile pretraining.

Figure 2. FreeTacMan links human demonstration, visual-tactile sensing, and robot execution.

The hardware includes collection and execution interfaces with matching visual and tactile observations. The system also provides a universal gripper interface with quick-swap mounts compatible with robot platforms such as Piper and Franka. This modularity matters because robot learning datasets are more useful when the data collection method can adapt across tasks and embodiments.

During collection, the system records wrist-camera RGB images, visual-tactile images, gripper width, and end-effector pose. Each trajectory becomes an embodiment-agnostic data sequence that links what the human operator sees, what the gripper feels, and how the gripper moves.

What role did NOKOV play in FreeTacMan?

In the FreeTacMan human-to-robot data transfer workflow, NOKOV provided high-precision optical motion capture for 6D pose tracking of the collection interface. The system tracked the handheld gripper's position and orientation, then synchronized the pose data with RGB images, visual-tactile images, and gripper width. This made it possible to convert human demonstration into a complete multimodal trajectory that could be mapped into robot execution.

Figure 3. FreeTacMan collection and execution interfaces with visual-tactile sensing and marker-based pose tracking.

Five retro-reflective markers were mounted on the interface: three on the top plate for overall pose and two on the gripper for relative displacement. With this marker setup, NOKOV measured how the collection interface moved in space while the operator performed contact-rich tasks.

For robot learning, motion capture provides more than position data. It provides the spatial reference that connects human demonstration with robot action representation. Without accurate pose tracking, the system may still record what the operator sees and feels, but it has a weaker description of how the gripper actually moved through space.

This is the measurement problem that NOKOV solved in the FreeTacMan experiment: high-precision position, orientation, and trajectory acquisition in a workflow where motion data had to align with visual, tactile, and gripper-state signals. In this pipeline, NOKOV helped turn human demonstration into synchronized, measurable robot training data.

What did the project results show?

The FreeTacMan dataset spans 50 contact-rich manipulation tasks, includes more than 10,000 manipulation trajectories, and contains over 3 million visuo-tactile image pairs. The project evaluates tasks such as fragile cup manipulation, USB plugging, texture classification, stamp pressing, and calligraphy.

The project also reports a user study with 12 human participants, comparing FreeTacMan with previous data collection setups such as ALOHA and UMI. FreeTacMan achieved strong completion rates and collection efficiency, and the project page defines CPUT, or Completion per Unit Time, as completion rate multiplied by efficiency. This metric matters because it combines whether the task can be completed and how quickly usable demonstrations can be collected.

For policy learning, the project page reports that imitation policies trained with FreeTacMan visuo-tactile data achieved an average success rate 50% higher than vision-only approaches across evaluated contact-rich tasks. This result reinforces the central point: for manipulation tasks where contact changes the outcome, tactile information and accurately synchronized motion data can make robot learning more robust.

FAQ

What is FreeTacMan?

FreeTacMan is a robot-free, human-centric visuo-tactile data collection system for contact-rich manipulation. It uses a handheld gripper with modular visual-tactile sensors to collect demonstrations for robot learning.

Why does contact-rich robot learning need tactile data?

Tactile data records local contact, deformation, texture, slip, and force-related interaction cues that are difficult to infer from vision alone. These cues are important for fragile grasping, insertion, stamping, writing, and other contact-sensitive tasks.

Why does this type of data collection need 6D pose tracking?

6D pose tracking records the position and orientation of the gripper or collection interface. It tells the learning system how the demonstrator moved through space, making the data more useful for robot execution and imitation learning.

How does motion capture help compared with SLAM or IMU tracking?

Optical motion capture can reduce drift and tracking error in controlled experimental environments. This makes it useful when pose data must synchronize accurately with RGB images, tactile images, and gripper-state data.

What did NOKOV measure in the FreeTacMan study?

NOKOV measured the 6D pose of the FreeTacMan collection interface and synchronized the pose data with RGB images, visual-tactile images, and gripper width. This provided the spatial trajectory layer needed for human-to-robot data transfer.

Source: FreeTacMan project page, OpenDriveLab, https://opendrivelab.com/FreeTacMan; Longyan Wu, Checheng Yu, Jieji Ren et al., "FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation", ICRA 2026.