Capturing Motion,
Crafting Stories

The research team led by Professor Lyu Ximin at Sun Yat-sen University has recently published their paper, "Whole-Body Integrated Motion Planning for Aerial Manipulators," in the prestigious journal IEEE Transactions on Robotics (T-RO).

The paper proposes a whole-body integrated motion planning framework for aerial manipulators. This framework features three key innovations:

(1) Dynamic Collision Volume Fitting: It fully considers the changes in collision volume caused by the configuration changes of the manipulator during actual operation, effectively addressing potential safety issues arising from these changes;

(2) Flexible Waypoint Constraint System: Utilizing various types of flexible waypoint constraints, it can flexibly combine to form various different operational sub-tasks;

(3) Imitation Learning-Guided Optimization: During deployment, the researchers found that existing algorithms often failed to meet task requirements for aggressive maneuvers (e.g., roll angle > 45°) as they frequently converged to poor suboptimal solutions. Through in-depth analysis of successful aggressive maneuver trajectories, they discovered that UAVs exhibit specific flight trends when executing such maneuvers (as shown in Figure 1(b) below). Building on this insight, they innovatively introduced an imitation learning prior to guide the optimization, ultimately achieving superior execution trajectories.

Additionally, this work thoroughly validated effectiveness of the framework and practicality through comprehensive ablation studies, as well as simulations and real-world experiments.

Paper Link:https://arxiv.org/abs/2501.06493

Project Page:https://github.com/SYSU-HILAB/am-planner

Figure 1:Demonstration of aerial striking using whole-body integrated motion planning. (a) The aerial manipulator executes a planned trajectory to reach a constrained waypoint, maintaining an attitude parallel to the inclined plane while configuring its delta arm for balloon striking. (b) Time-lapse visualization capturing the complete execution of the planned motion.

Research Background

Aerial manipulators integrate the rapid mobility of UAVs with the dexterous manipulation capabilities of robotic arms, enabling the execution of diverse tasks in 3D space. However, most existing algorithms and hardware for aerial manipulators are task-specific, typically limited to predefined single tasks such as grasping or writing. For manipulators, there is a strong demand to support diversified operational tasks. How to characterize the planning of such varied tasks within a unified framework has become a key challenge in this field.

Active Flight Corridor Generation Strategy

To support the implementation of this framework and ensure adequate solution space around all waypoint constraints, this study first introduces an enhanced flight corridor generation strategy. By establishing a set of generation rules, it actively generates two polyhedra near each constrained waypoint, which are then connected with all other polyhedra through conventional flight corridor construction. Figure 2 illustrates the active flight corridor generation method, while Figure 3 presents an ablation study on flight corridor generation, demonstrating that the proposed method guarantees sufficient solution space near waypoint constraints while conforming closely to the inclined surface.

Figure 2: Active Flight Corridor Generation Method.

Figure 3: Ablation Study on Flight Corridor Generation. The results demonstrate that the proposed method ensures sufficient solution space near the waypoint constraints while conforming closely to the inclined surface.

Dynamic Collision Volume Fitting

Regardless of the specific task, this work must also ensure that the aerial manipulator possesses fundamental flight capabilities, generating trajectories that are smooth, efficient, and satisfy dynamic and kinematic feasibility constraints, as well as safety constraints. The most prominent feature of this work lies in its optimization of safety constraints. Considering that the manipulator is in constant motion during actual operation, the collision volume of the entire aerial manipulator system continuously changes. Traditional obstacle avoidance planning for UAVs employs static ellipsoids or spheres to approximate the collision volume for subsequent optimization; however, these methods cannot be directly applied to aerial manipulators. Excessively aggressive approximations would lead to safety issues, while overly conservative ones would impair the operational capability of the UAV.

This work innovatively proposes a varying ellipsoid fitting approach that adjusts the dimension of the ellipsoid along the z-axis based on the extension length of the manipulator. This enables the aerial manipulator to retract its arm when necessary and extend it during task execution, as shown in Figure 4 below.

Figure 4: Collision Volume Approximation Using a Varying Ellipsoid for the Aerial Manipulator.

Flexible Waypoint Constraint System

To accommodate diverse task requirements, this work innovatively proposes three types of waypoint constraints for aerial manipulators:

• Point Constraint: Requires either the quadrotor or the end-effector to pass through a specific point in the world frame. Examples include the end-effector vertically contacting an object during grasping, the end-effector penetrating a balloon during striking, or the quadrotor traversing the center point of a gate.

• Axis-Wise Constraint: Constrains the end-effector to move linearly along a specific axis, such as pushing an object forward along the x-axis.

• Plane-Wise Constraint: Requires the end-effector to remain within a designated plane, such as pulling an object across a table surface or writing on a planar surface. 

Figure 5: Decomposition of different subtasks through waypoint constraints.

Imitation Learning-Guided Optimization

When flying under constrained conditions for aggressive maneuvers, if the initial solution is poor, it may lead to a final local optimum that fails to meet the required conditions. As shown in Figure 6(a), the orange trajectory of the robotic arm completely misses the target point (red point). However, by analyzing these successful aggressive maneuver trajectories, this work discovered that UAVs tend to follow "arc-shaped" trajectories to satisfy dynamic constraints. Therefore, by collecting a large number of effective trajectories, imitation learning can be used to fit the trends of these local trajectories. But how can these trends be effectively utilized?

Based on the aforementioned waypoint constraint handling method, this work samples several guide points from these local trajectories to enforce the trajectory to pass through them. However, excessively strict guide point constraints may still result in poor trajectories or even failure. To address this, the work innovatively proposes applying guide point constraints during the initial warm-up optimization stage and removing them after the trajectory is roughly shaped to obtain better trajectories. Comprehensive ablation studies were conducted on polynomial learning versus discrete point learning, the number of guide points, and the necessity of the second optimization stage. The final results, shown in Figures 6(b-f) and 7, demonstrate that the two-stage optimization employing polynomial learning with two guide points achieves the best performance.

Figure 6: Ablation study on guided optimization, comparing two learning methods (polynomial trajectories and discrete points) with varying numbers of guide points.

Figure 7:Convergence curves comparison across optimization methods under randomly sampled scenarios.

Simulation Validation

To validate the effectiveness of the framework, this work tested the performance of three different manipulator configurations in grasping tasks through simulations:

• Telescopic manipulator: motion constrained solely to the z-axis

• 1-DOF manipulator: capable of moving along a circular arc

• 2-DOF manipulator: enables planar motion within a single plane

As shown in Figure 8, the end-effector positions remain effectively constrained within their respective workspaces, and all configurations successfully accomplish the task.

Figure 8: End-effector motion of aerial manipulators with different configurations in grasping tasks.

Real-world Implementation

This work evaluated nine fundamental manipulation skills: strike, grasp, push, pull, lift, press, wind, cross, and write. Among these, both strike and grasp were executed under large-attitude manipulation constraints and additionally employed Imitation Learning-Guided Optimization.

NOKOV Motion Capture System provided high-precision real-time pose data for the UAV and targets in this study, enabling more accurate control and validating the proposed methodology.

About Lyu Ximin

Ximin Lyu: He is an Associate Professor supported by the "Hundred Talents Program" at Sun Yat-sen University. He earned his Bachelor and Master degrees from Harbin Institute of Technology, and received his Ph.D. in Electronic and Computer Engineering from The Hong Kong University of Science and Technology in 2019. From 2018 to 2021, he served as a Senior Control Algorithm Researcher at DJI Technology Co., Ltd.

In recent years, he has been dedicated to algorithm research and practical applications related to unmanned aerial vehicles, having published over 20 high-quality papers in prestigious journals and conferences including Science Robotics, T-RO, RAL, ICRA, and IROS. He has filed more than 20 international patents (PCT) and Chinese national invention patents, with the majority already widely implemented in industrial products.

He has led or participated in several key research initiatives, including the Young Scientists Program of the National Key Research and Development Program of China, the Young Scientists Fund of the National Natural Science Foundation of China, Guangdong-Hong Kong Joint Funding Program for Technological Innovation, the Shenzhen Outstanding Talents Project, as well as multiple defense-related and industry collaboration projects.