DMAD – Diffusion Model Adversarial Distillation Framework Jointly Open-Sourced by ByteDance and Texas A&M University

1. What is DMAD

DMAD (Distribution Matching as Adversarial Distillation) is an open-sourced few-step distillation framework jointly developed by ByteDance and Texas A&M University. Its core objective is to significantly accelerate the generation process of diffusion models. This method trains a binary classifier to directly learn the differences between the target distribution and the student distribution, eliminating the reliance on auxiliary diffusion models in traditional distillation approaches. On the ImageNet-64 dataset, DMAD achieves one-step generation results with an FID as low as 1.04, efficiently compressing a 50-step teacher model into a student model that requires only 4 steps. DMAD now supports multiple mainstream models, including ImageNet, SDXL, Wan2.1, and MiniMax-H3-33B, covering image, video, and joint audio-video generation scenarios. It can generate a 1344×768 audio video of approximately 5 seconds on consumer-grade GPUs with 8GB VRAM, significantly lowering the technical barriers for high-quality content generation.

Technical positioning and domain: DMAD belongs to the interdisciplinary research area of generative model acceleration and knowledge distillation, focusing on optimizing the sampling efficiency of diffusion models. The proposed "distribution matching as adversarial distillation" paradigm transforms the few-step sampling problem of generative models into an adversarial learning problem for a binary classifier, providing a new technical pathway for the efficient deployment of diffusion models.

Development background: This framework was jointly developed by ByteDance and Texas A&M University. The team observed that traditional distillation methods rely on the continuous fitting of auxiliary score models, resulting in significant computational redundancy and training overhead. Therefore, they designed a more direct distribution matching approach. This work was completed under the joint efforts of academia and industry, balancing theoretical innovation with engineering practicality.

Core value: DMAD addresses the issue of high computational costs caused by multi-step iterative sampling during the inference phase of diffusion models. By compressing the sampling steps from 50 to 4 or even 1, it achieves over 92% reduction in denoising iterations while maintaining generation quality close to that of the teacher model. Its unified framework design allows image, video, and audio-video generation tasks to share the same distillation mechanism. Additionally, training efficiency is improved by approximately 3.51 times under a 4-H200 GPU configuration, providing a feasible validation for the efficient deployment of large-scale generative models.

Technical features: The method employs a discriminator with a shared feature backbone and dual discrimination heads, enabling distinction learning between teacher samples and real data samples. It also utilizes a dynamic weighting mechanism for noise intervals to adaptively adjust the weights of the two types of supervision signals. Furthermore, the offline teacher sample caching strategy eliminates the online dependency on the teacher model during training, significantly improving training throughput.

2. Key Features

• Ultra-fast, Few-step Image Generation: DMAD can distill the EDM model from ImageNet-64 into a single-step sampling process, achieving an FID as low as 1.04, significantly outperforming existing methods such as DMD2 (1.28). The distilled SDXL model only requires 4 steps to complete high-quality text-to-image tasks, reducing inference latency by an order of magnitude while maintaining visual quality.

• Few-step Text-to-Video Generation: This framework successfully distills the large-scale video generation model Wan2.1-T2V-14B into a 4-step student model, achieving a VBench score of 85.15. This demonstrates the effectiveness of few-step distillation in high-dimensional and complex tasks like video generation, with good preservation of temporal consistency in the video.

• Joint Audio-Video Generation: DMAD distills the 50-step teacher model of MiniMax-H3-33B into a 4-step student version, supporting direct output of video with a resolution of 1344×768 and 32kHz native stereo audio. This enables the joint generation of visual and auditory signals, expanding the application boundaries of distillation techniques.

• Consumer-grade Hardware Friendly: The project provides ComfyUI workflows optimized for 8GB VRAM and low VRAM command-line inference scripts. Practical tests show that it can generate approximately 5 seconds of audio-video content on an 8GB VRAM GPU, and up to about 15 seconds on a 12GB VRAM GPU, allowing ordinary creators to access advanced generative AI capabilities without requiring high-end GPUs.

• Dynamic Noise Interval Weighting: To address the distribution differences between teacher samples and real data that vary with noise levels, DMAD uses the average logit difference of real data heads across the two types of samples as an empirical signal, tracking it using an exponentially weighted moving average with a decay factor of 0.99. Noise intervals with smaller distribution gaps automatically receive higher teacher weights, which are normalized and incorporated into the student loss calculation, thereby enhancing training stability.

• Offline Teacher Sample Acceleration: The teacher model only needs to pre-generate and cache samples before training, with no online calls required during the training process. Under a 4-GPU H200 configuration and MiniMax-H3 rank-128 LoRA setup, the time required for each student update and related discriminator update decreases from 818.1 seconds to 233.3 seconds, achieving approximately 3.51 times faster training.

• Fully Open-source for Training and Inference: The project releases the complete implementation of the distillation method, training code, inference scripts, and LoRA weights for multiple models, covering both command-line and ComfyUI usage methods. Developers can reproduce the distillation process based on the open-source code or directly deploy inference using existing weights.

3. How to Use

1. Clone the repository: Execute git clone https://github.com/Yzmblog/DMAD.git to obtain the project code and navigate to the project root directory. It is recommended to use the Linux operating system for the best compatibility.

2. Set up the environment: Create a Python 3.12 environment using conda and activate it. Recommended command: conda create -n dmad python=3.12, followed by conda activate dmad.

3. Install dependencies: Install PyTorch 2.11.0 (cu126) and its corresponding torchvision and torchaudio packages using the official fixed versions. After installing the base framework, run pip install -r requirements.txt to install the remaining project dependencies, and then execute pip install -U modelscope-hub to ensure version compatibility of the model download components.

4. Prepare the hardware environment: Ensure the host is equipped with an NVIDIA GPU that supports CUDA, and reserve approximately 64GB of system memory and 170GB of disk space for model weight storage and temporary inference files. The GPU memory capacity directly determines the length of the video that can be generated; a minimum of 8GB is required, with 12GB or more providing a better experience.

5. Download the base model: Download the MiniMax-H3 base model from the ModelScope platform using the command modelscope download MiniMax/MiniMax-H3. This model provides the teacher model weights, used in conjunction with LoRA for few-step distillation inference.

6. Download the distillation LoRA weights: Obtain the official 4-step distillation LoRA weight files released by DMAD from the ModelScope community, and ensure the path configuration is correct.

7. Run command-line inference: Execute the inference.py inference script, specifying the model path, LoRA path, and prompt file in sequence. Set the --steps 4 parameter to select the 4-step sampling mode. Set the resolution parameters to --height 768 --width 1344, and choose the --low-vram 8 or --low-vram 12 mode for GPU memory management based on your hardware capabilities.

8. View the output results: After inference is complete, the output directory will contain the 0000.mp4 video file, a corresponding txt file for the prompts, and a settings.json configuration record. Once the successful run is verified, you can replace the prompt file with your own to generate videos in bulk.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

In specific selection scenarios, if researchers are focused on theoretical innovation in image distillation and improvements in training efficiency, DMAD's offline teacher caching and dynamic weighting mechanism offer clear advantages. Additionally, its relatively complete open-source resources make it suitable as a baseline framework for further development. For creators who only need fast image generation on consumer-grade hardware, DMD2's SDXL support is relatively mature and well-documented, making it a good entry-level choice.

If the project involves video or audio-visual joint generation, DMAD is currently the most comprehensive open-source solution in the few-step distillation domain. The pre-trained LoRA weights for Wan2.1 and MiniMax-H3 allow researchers to validate few-step generation effects without training from scratch. For teams pursuing top-tier image quality, with ample computational resources and the ability to tolerate longer training cycles, SiDA's alternative technical approach can serve as a comparative benchmark for cross-validation.

6. Editor's Summary

DMAD proposes a technically innovative approach in the field of diffusion model distillation. Traditionally, few-step distillation relies on auxiliary diffusion models to continuously fit the student's score function, a process that introduces additional training complexity and computational overhead. DMAD transforms the distribution matching problem into an adversarial training problem of a binary classifier through theoretical derivation, leveraging the relationship between the logit of the optimal discriminator and the log density ratio to directly obtain the correction direction for the student model. This approach simplifies the distillation framework, and the FID 1.04 results achieved on ImageNet-64 provide strong empirical evidence.

In terms of practical value, DMAD reduces the number of steps from 50 to 4 while cutting training costs by approximately 3.51 times. The simultaneous optimization of these two metrics holds significant implications for industrial deployment. Its ability to generate 5 to 15 seconds of audio-video content on consumer-grade GPUs effectively lowers the entry barriers for high-quality generative content creation devices. The unified framework covers multiple modalities, including image, video, and audio, reducing engineering costs associated with redundant development.

In terms of target users, DMAD is primarily aimed at three categories: academic teams researching acceleration theories for diffusion models, who can use this framework as a baseline for comparison or as a starting point for improvements; AI application developers requiring efficient deployment of generative models, who can directly use its open-source weights and inference scripts; and individual creators relying on generative AI for content production, who can achieve low-threshold usage through the ComfyUI workflow.

Overall, DMAD demonstrates strong completeness in terms of theoretical innovation, engineering implementation, and ecosystem development. Future adaptability of its framework across more model architectures, as well as further expansion of its multi-modal joint generation capabilities, are worth continued attention.

7. Application Scenarios

• Short Video Content Creation: Supports generating 5 to 15-second video clips with native audio in just 4 steps, enabling creators to quickly produce social media content on 8GB consumer-grade GPUs. Whether for product showcases, scene transitions, or story snippets, DMAD significantly reduces production time while maintaining quality, lowering content production costs for small teams.

• Advertising and Marketing Material Generation: Brands can generate product demonstration videos and scenario-based marketing clips in bulk based on text prompts, combined with 32kHz high-quality audio output, to rapidly build an advertising asset library. Short video ads often require frequent iterations to test different creative directions, and the streamlined distillation process enables fast generation and screening of multiple versions, effectively shortening the cycle from concept to deployment.

• Film Previs and Concept Design: Directors and art teams need to quickly validate camera language and visual atmosphere during the early stages of a project. DMAD can generate camera previsualizations, scene concept art, and storyboard drafts in a short time, assisting in creative discussions and decision-making. The efficient generation process from streamlined distillation enables real-time iteration, accelerating the exploration of early-stage creative ideas.

• Game and Animation Asset Production: Independent game studios and small animation teams can use DMAD to generate character animations, scene concept art, and dynamic background assets, acquiring high-quality visual assets at a lower cost. In projects with limited budgets and manpower, this tool helps content teams validate a wider range of visual concepts with fewer resources.

8. FAQ

Q: What is the core difference between DMAD and traditional diffusion model distillation methods?
A: Traditional methods like DMD2 require a teacher score and an auxiliary diffusion model to continuously fit the student's score function, resulting in a complex training process and high computational costs. DMAD instead trains a binary classifier to directly distinguish between target samples and student samples, leveraging the relationship between the optimal discriminator's logit and the log density ratio to directly obtain the correction direction for the student model. This eliminates the need for an auxiliary diffusion model and significantly simplifies the training process.

Q: What is the minimum hardware configuration required to run DMAD inference?
A: The officially provided low-memory mode supports NVIDIA GPUs with 8GB VRAM. Under this configuration, it can generate a 1344×768 resolution audio-video clip of about 5 seconds. With 12GB VRAM, the generation duration can be extended to approximately 15 seconds. It is recommended to reserve 64GB of system memory and approximately 170GB of disk space for model weights and runtime files.

Q: Which base models and generation tasks does DMAD support?
A: Currently, DMAD supports the EDM model for ImageNet-64 for few-step image generation, SDXL for text-to-image generation, Wan2.1-T2V-14B for text-to-video generation, and MiniMax-H3-33B for joint audio-video generation. These four models correspond to three types of generation tasks: image, video, and audio-video. Distilled LoRA weights are provided for all of them.

Q: Can I use my own dataset or model for DMAD distillation training?
A: Yes. The project has open-sourced complete distillation training code, supporting the distillation process with custom models and datasets. Users need to prepare pre-generated sample caches from the target teacher model and configure the corresponding discriminator structure and training hyperparameters. A full training process still requires a multi-GPU high-performance cluster.

Q: How can DMAD be used in ComfyUI?
A: The project provides a dedicated ComfyUI workflow. Users simply need to import the workflow file from the repository into ComfyUI, configure the base model path and LoRA weight path, and can then perform few-step distillation inference through a node-based interface without needing to write command-line parameters. This makes it suitable for creators without a technical background.

9. Project Links

• Project Website: https://yzmblog.github.io/projects/DMAD/
• GitHub Repository: https://github.com/Yzmblog/DMAD
• HuggingFace Model Library: https://huggingface.co/ZhengmingYu/DMAD
• arXiv Technical Paper: https://arxiv.org/pdf/2610.02188