Chitu

Chitu

by Tsinghua University, Qingcheng Jizhi
Chitu is a high-performance large model inference engine designed to reduce costs and improve efficiency during the inference phase, supporting multiple NVIDIA GPUs and domestic chips.

What is Chitu?

Chitu is a high-performance large model inference engine jointly open-sourced by Tsinghua University's High-Performance Computing Institute and Qingcheng Jizhi. It is designed to address the high costs and inefficiencies of large models during the inference phase. Chitu has strong hardware adaptability, supporting multiple NVIDIA GPUs and domestic chips, breaking the dependency on specific hardware like NVIDIA's Hopper architecture.

Main Features of Chitu

  • Diverse Computing Power Adaptation: Supports multiple series of NVIDIA GPUs, from the latest flagship to older models, while also providing optimized support for domestic chips.
  • Full-Scenario Scalability: From pure CPU deployment, single GPU deployment to large-scale cluster deployment, Chitu provides scalable solutions.
  • Low Latency Optimization: Optimizes model inference speed for latency-sensitive scenarios such as financial risk control.
  • High Throughput Optimization: Increases the number of requests processed per unit time in high-concurrency scenarios such as intelligent customer service.
  • Small Memory Optimization: Reduces single-card memory usage, allowing enterprises to achieve higher inference performance with fewer hardware resources.
  • Long-Term Stable Operation: Chitu can be used in actual production environments, with stability sufficient to handle concurrent business traffic.
  • Out-of-the-Box: Qingcheng Jizhi has launched an inference all-in-one machine based on Chitu, providing out-of-the-box deployment solutions and professional operation and maintenance services.

Technical Principles of Chitu

  • Underlying Technological Innovation: Chitu achieves native operation of FP8 precision models on non-NVIDIA Hopper architecture GPUs and various domestic chips.
  • Operator-Level Optimization: Chitu has performed instruction-level optimization on key operators, directly processing FP8 data without losing model accuracy.
  • Full-Scenario Performance Optimization: Chitu supports low latency, high throughput, and small memory optimization, providing optimal solutions based on different scenario requirements.
  • Parallel Computing and Compilation Optimization: Chitu leverages years of parallel computing and compilation optimization technology accumulated by the Tsinghua University team.

Application Scenarios of Chitu

  • Risk Identification and Early Warning: Quickly processes massive transaction data, monitors potential risks in real-time, and provides timely warnings.
  • Intelligent Customer Service and Customer Experience Optimization: Through large model intelligent knowledge bases, Chitu can quickly respond to customer needs.
  • Disease Diagnosis Assistance: Quickly processes medical data, improving the speed and accuracy of disease diagnosis.
  • Traffic Flow Optimization: Processes traffic data in real-time, optimizing traffic flow and alleviating urban congestion.
  • Scientific Research Data Analysis: Efficiently processes scientific research data, accelerating the research process.

Framework Features

Supported Tasks
Inference Model Deployment Performance Optimization
Tags
Inference Engine Large Models Hardware Optimization AI Framework GPU Acceleration Performance Optimization Open Source Machine Learning High Performance Computing AI Deployment

Getting Started

Pricing
free

Screenshots & Images

Additional Images
View Repository

Stats

0 Views
0 Favorites
1069 GitHub Stars

Community & Support

GitHub Repository

Similar Frameworks

TPO
0
Phantom by ByteDance
0
AgentSociety by Tsinghua University
0

Recently Viewed

BEHAVIORRobotSuite Framework