MimicMotion - Tencent's AI Human Motion Video Generation Framework

What is MimicMotion?

MimicMotion is a high-quality human motion video generation framework developed by Tencent researchers. It uses confidence-aware pose guidance technology to ensure high-quality video frames and smooth temporal transitions. Additionally, MimicMotion reduces image distortion and enhances hand movement details through region loss amplification and hand region enhancement. The framework can generate long videos with high quality and temporal coherence using a progressive latent fusion strategy, significantly improving control and detail richness in video generation.

Features of MimicMotion

• Generate Diverse Videos: MimicMotion can generate various motion videos based on user-provided pose guidance. Whether it's dance, sports, or daily activities, MimicMotion can create corresponding dynamic videos.
• Control Video Length: Users can specify the duration of the video, allowing MimicMotion to generate anything from short clips of a few seconds to longer videos of several minutes or more.
• Pose Guidance Control: The framework uses reference poses as conditions to ensure the generated video content matches the specified poses, allowing for precise control over the video's actions.
• Detail Quality Assurance: MimicMotion pays special attention to details in the video, especially in areas prone to distortion like the hands, providing clearer visual effects through confidence-aware strategies.
• Temporal Smoothness: To offer a more natural viewing experience, MimicMotion ensures smooth transitions between video frames, avoiding stuttering or incoherence.
• Reduce Image Distortion: Through confidence-aware pose guidance, MimicMotion can identify and reduce image distortion caused by inaccurate pose estimation, especially in the hand regions.
• Long Video Generation: MimicMotion uses progressive latent fusion technology to maintain high temporal coherence when generating long videos, effectively avoiding flickering and incoherence.
• Resource Consumption Control: The framework optimizes algorithms to keep resource consumption within reasonable limits, even when generating longer videos.

Official Links

• Official Project Homepage
• GitHub Source Code Repository
• arXiv Technical Paper

Technical Principles of MimicMotion

• Pose-Guided Video Generation: MimicMotion uses user-provided pose sequences as input conditions to guide video content generation, allowing the model to synthesize corresponding actions based on pose changes.
• Confidence-Aware Pose Guidance: The framework introduces the concept of confidence, weighting each key point in the pose sequence based on the confidence score provided by the pose estimation model. This allows the model to rely more on key points with higher confidence, reducing the impact of inaccurate pose estimation.
• Region Loss Amplification: Specifically for areas prone to distortion like the hands, MimicMotion increases the weight of these regions in the loss function, enhancing the model's training on these areas and improving the quality of hand details in the generated video.
• Latent Diffusion Model: MimicMotion uses a latent diffusion model to improve generation efficiency and quality. The model operates in a low-dimensional latent space rather than directly in the pixel space, reducing computational costs.
• Progressive Latent Fusion: To generate long videos, MimicMotion employs a progressive latent fusion strategy. By gradually fusing the latent features of overlapping frames between video segments, it achieves smooth transitions and avoids flickering and incoherence.
• Pre-trained Model Utilization: MimicMotion is based on a pre-trained video generation model (such as Stable Video Diffusion, SVD), reducing the data and computational resources required for training from scratch.
• U-Net and PoseNet Structure: MimicMotion's model structure includes a U-Net for spatiotemporal interaction and a PoseNet for extracting pose sequence features. These networks work together to achieve high-quality video generation.
• Cross-Frame Smoothness: MimicMotion considers the temporal relationship between frames during the generation process, ensuring coherence and smoothness between video frames.