What is Tora?

Tora is an AI video generation framework developed by Alibaba, based on the Trajectory-guided Diffusion Transformer (DiT) technology. It integrates text, visual, and trajectory conditions to produce high-quality videos that align with real-world physical dynamics. Tora consists of a Trajectory Extractor, Spatial-Temporal DiT, and Motion-guidance Fuser, enabling precise control over video dynamics and supporting video production of up to 204 frames at 720p resolution. Tora excels in motion fidelity and simulating real-world physical dynamics, offering a powerful new tool in the field of video generation.

Main Features of Tora

In simple terms, Tora can create realistic and smooth videos based on your instructions, such as text descriptions, images, or object movement paths.

• Trajectory Extractor (TE): Converts input trajectories into hierarchical spatiotemporal motion blocks that match the latent space of the video content.
• Spatial-Temporal DiT: Combines spatial and temporal self-attention mechanisms to process video data, enabling the model to understand and generate videos with coherent motion.
• Motion-guidance Fuser (MGF): Integrates the spatiotemporal motion blocks generated by the Trajectory Extractor into the DiT model, ensuring that the generated video content follows the predetermined trajectory and dynamics.

Technical Principles of Tora

• Trajectory Understanding: Tora uses a tool called the "Trajectory Extractor" to understand the given trajectory information. It's like giving Tora a map, telling it where and how objects should move in the video.
• Spatial-Temporal Encoding: Tora converts this trajectory information into a special encoding form called "spatiotemporal motion blocks." These motion blocks act as the skeleton of the video, determining how objects move.
• Video Generation Framework: Tora employs an advanced technology called the "Diffusion Transformer" (DiT), which combines the advantages of diffusion models and transformer architectures, allowing Tora to generate high-quality videos.
• Dynamic Fusion: Tora also includes a "Motion-guidance Fuser," which integrates the spatiotemporal motion blocks with the video content. This ensures that the generated video not only looks good but also has natural and smooth object movements.
• Two-Stage Training: To better understand and generate motion, Tora undergoes a two-stage training process. First, it learns how to extract motion information from dense optical flow (a type of dense data describing object motion). Then, it learns how to generate videos based on simpler trajectory information provided by the user.
• Data Preprocessing: Before training, Tora processes video data by segmenting long videos into shorter clips based on scene detection, and then selecting suitable video clips for training based on aesthetic scores and motion segmentation results.

Project Addresses of Tora

• Project Website: https://ali-videoai.github.io/tora_video/
• GitHub Repository: https://github.com/ali-videoai/Tora
• arXiv Technical Paper: https://arxiv.org/pdf/2407.21705

Application Scenarios of Tora

• Film and Television Production: Tora can be used to generate special effects scenes in movies, TV shows, or short films, reducing the cost and time of actual shooting by controlling trajectories to create complex dynamic scenes.
• Animation Creation: In the field of animation, Tora can automatically generate animation sequences based on scripts, providing animators with preliminary dynamic sketches and speeding up the creative process.
• Virtual Reality (VR) and Augmented Reality (AR): Tora can generate dynamic environments that interact with users, providing realistic visual effects for VR and AR applications.
• Game Development: In video games, Tora can be used to quickly generate game environments and character animations, improving the efficiency of game design.