Announcing the Open-Source Release of ML Video Codec (MLVC)

Video codecs compress video for transmission or storage, reducing bandwidth and storage requirements. MLVC is a modern machine-learning-based codec that uses substantially less bandwidth than conventional codecs, improving streaming and video-call quality—especially on constrained or unreliable networks—while lowering delivery and storage costs.

MLVC is the product iteration of DCVC (Deep Contextual Video Compression) family of NVC (Neural Video Codec), open sourced by Microsoft Research since 2021, with improved compression efficiency, real-time performance on commodity Neural Processing Units (NPUs), and cross-platform support. We recently published this work in the paper MLVC: Multi-platform Learned Video Codec for Real-World Deployment. We are releasing the source code because we believe the next generation of video coding will be built openly, and we want the broader community — researchers, video codec engineers, platform vendors, product teams, as well as general developer community — to build it with us.

Why MLVC

Traditional video codecs (e.g., H.264/AVC, H.265/HEVC) have served the industry for a long time, but each generation requires enormous engineering effort for incremental gains and needs dedicated hardware which takes years to become commonly available. MLVC replaces conventional primitives — motion estimation, transforms, entropy modeling — with end-to-end learned neural compression, trained directly against rate-distortion objectives, and run on general-purpose NPU devices. 

The table below compares MLVC to popular video codecs, showing its lower bitrate and resulting savings in bandwidth and storage.

For example, for 360p video at 30 fps, where H.264 requires 1 Mbps, MLVC requires roughly 122 kbps for equivalent quality — about one-eight the bitrate under real-time conditions. The inference compute was kept approximately equal for the 360p and 540p resolutions. These results are based on a P.910 subjective test and are based on the Video Conferencing Dataset (VCD) dataset that we developed and also recently released as an open-source project.

The following video demo illustrates the extent of quality enhancement achieved by MLVC relative to H.265/HEVC at the same bitrate of 200kbps (please watch by opening in a new window for better demonstration)

Beyond video compression efficiency, MLVC also offers:

• NPU-first design. MLVC is built to run almost entirely on the AI accelerators already shipping in modern devices — Apple Neural Engine, Qualcomm and Intel NPUs — at no more than 50% NPU utilization, leaving NPU headroom for the rest of the system.
• Real-time execution at the targeted operating points. Demonstrated 540p at 30 fps on Apple, Intel, and Qualcomm hardware.
• A scaling-law trajectory. Empirically, MLVC's coding efficiency improves with increased model capacity and additional training compute.
• Content-adaptive behavior out of the box, without hand-tuned Rate Distortion Optimization heuristics.

Already running in Microsoft Teams

MLVC is more than just a research concept for Microsoft. We are currently rolling it out in Microsoft Teams, where it is being validated on real peer-to-peer video calls with active telemetry and A/B testing. The integration runs alongside fallback to conventional video codecs for hardware or reliability constraints — the kind of mixed-environment deployment that real products need. The scaling and reliability insights from this rollout are shaping the codec and its roadmap.

We welcome your contributions

We can not cover every use case, every device class, or every content domain by ourselves. That is why we are open sourcing MLVC. If you work on:

• Streaming, Video On Demand (VOD), or live broadcast
• Real-time communication and conferencing
• Cloud gaming or remote rendering
• Surveillance, drones, or robotics
• Mobile capture, Augmented Reality (AR) / Virtual Reality (VR), or volumetric video
• Codec hardware, NPUs, or inference runtimes

We would love your help. Contributions of all kinds are welcome, including model improvements, training recipes, new platform ports and conversion targets, runtime backends, domain-specific fine-tunes, evaluation tooling, bug reports, and feedback on what is missing for your scenario. We particularly welcome platform ports that expand NPU coverage and efficiency improvements that push the rate-distortion frontier.

What's in the release

The MLVC repository is available at https://github.com/microsoft/mlvc and shared under the MIT License. It includes:

• MLVC model source code of the full network architecture.
• Trained model weights ready to run.
• Training scripts used to produce shipping models.
• Training data collection documentation to help reproduce and improve MLVC.
• Platform conversion scripts to target different NPUs and runtimes.

Issues and pull requests will be open from day one.

A follow-up release will add a C++ codec library, simplifying integration into real-world applications.

Where we're heading

Our long-term goal with MLVC is to create an open, learned video codec that meets or exceeds the coding efficiency of the best conventional codecs across the full range of video content, runs efficiently on the AI hardware already shipping in client and cloud devices, scales with compute the way modern ML systems do, and evolves in the open at the pace of the ML community rather than the pace of standardization cycles.

In the near term that means stabilizing 540p real-time performance, expanding hardware coverage, and improving loss resilience. In the medium term: higher resolution, e.g., 1080p, and broader streaming scenarios. In the long term: an open video codec ecosystem that meaningfully replaces legacy stacks where it makes sense to.

We can't create the future of MLVC alone. We're glad you're here, and we are looking forward to building the next-generation video codec with you.

Who are we

MLVC is brought to you by the following awesome folks working on the project at Microsoft: Ross Cutler, Ando Saabas, Tanel Pärnamaa, Ardi Loot, Haiyan Xie, Lauri Ehrenpreis, Andrei Znobishchev, Martin Lumiste, Evgenii Indenbom, Yan Lu, Bin Li, Jiahao Li, Naba Kumar, Babak Naderi, Juhee Cho, Badal Yadav, Jinxin Zhou, Tianyu Ding, Patrick Gregory.

— The MLVC Team, Microsoft