Virtual live broadcasting displays virtual scenes and characters in real-time to the audience, replacing the original, more costly physical sets and live appearances with virtual counterparts. This provides viewers with a novel visual experience while also reducing the costs associated with set construction. Virtual live broadcasting has emerged as a new integrated technology following the advent of AI, VR, and motion capture technologies. The technology has seen rapid development, especially after the onset of the pandemic.

In the process of virtual live broadcasting, real human movements and expressions are displayed in real-time as virtual avatars. This involves the application of motion capture technology. So, how is this technology applied in virtual live broadcasting.

NOKOV motion capture has achieved high integration with the VUP virtual live broadcasting tool. By affixing reflective markers to the broadcaster, the optical motion capture system captures the coordinates of these markers. The information from multiple points is used to determine the pose data of rigid bodies, allowing for the creation and recognition of broadcaster models. Once the pose data is imported into third-party software and bound to a model, the system can produce effects for film, games, and virtual live broadcasts.

This article will be divided into three steps, introducing the implementation process of applying motion capture to virtual live broadcasting from the perspective of specific operational procedures.

Marker Placement

Markers, also known as reflective markers, are reflective spheres placed on objects being captured. They reflect the infrared light emitted by the motion capture cameras to obtain their 3D coordinate information.

When placing markers, the following points should be noted: ① Marker size, for general field human motion capture 12-15mm markers are recommended; ② Marker fixation method, the markers need to be firmly attached to the human body surface, and when wearing a motion capture suit, Velcro can be used for secure fixation; ③ Marker placement position, when establishing a virtual live broadcast system, professionals typically provide standard placement templates.

When designing on your own, here are several considerations: 1) Ensure that the points affixed are on the external side of the bones, easily recognizable; 2) Attach points at bony landmarks and joints; 3) Use multiple points to define a rigid body.  

Building Skeletal Systems

       After acquiring point cloud data on models, software processing can establish the model's skeletal system. This generates specific information templates from a set of reflective markers, describing points on the model, the lines between these points, and the rigid bodies defined by the points, termed Markerset.

Optical motion capture systems directly acquire only the precise coordinates of each marker point. Therefore, when creating a skeleton, it is necessary to ensure that the lengths of the bone segments correspond to the distances between their respective points, and their positions are also well aligned. This allows the points to accurately drive the motion of the skeleton. By importing the bone data from the Markerset into other software, various models can be animated.

Connection to Virtual Live Broadcasting Software

       Different models are available for selection in VUP, a virtual live broadcasting software. Once an appropriate model is selected, the virtual character can begin to mimic the actions of the live host. By connecting a smartphone or computer camera, the VUP software can capture the host's facial expressions in real-time; wearing data gloves, VUP can capture the detailed movements of the fingers, thereby facilitating interaction with the audience.

Additionally, the NOKOV optical motion capture system supports multi-person motion capture, making it possible to present multiple virtual characters on the same stage.

VUP virtual broadcasting is a further expansion of NOKOV's motion capture business area. NOKOV's self-developed optical motion capture system offers high cost-performance, with precise and stable data collection, providing strong technical support for virtual broadcasting.