Netflix Tackles AI Video Editing Challenges

Netflix is pushing the boundaries of creative tooling with early research into AI video editing. The streaming giant aims to empower storytellers by developing generative AI that offers granular control over complex visual effects, a significant step beyond current tools that often struggle with unintended alterations.

Promotional assets like trailers and social clips demand intricate edits, from seamless visual element integration to object removal, tasks that traditionally consume extensive manual labor. Current AI models often regenerate entire video frames, risking the integrity of original footage by inadvertently changing untouched elements. Netflix's research, detailed on netflixtechblog.com, seeks to address these limitations.

The core challenge lies in creating AI that respects creative intent. Many generative AI for video editing approaches regenerate every pixel, leading to issues like unnatural physics or altered identities. This research focuses on ensuring AI serves, rather than dictates, the artist's vision, building on advancements in the field of generative AI for video editing.

Vera: Layered Video Editing

Vera, a layered video diffusion model, tackles the problem of unintended edits. Instead of regenerating the entire video, Vera generates changes as separate edit layers. This approach ensures that pixels outside the edited regions remain precisely as filmed, preserving original identities and performances.

The model works by jointly generating an edit layer and an alpha matte. These are then composited with the source footage, allowing for tasks like object addition and background replacement without disturbing the original scene's integrity.

Developing Vera required a custom dataset of 486k frames, built from open-source videos and human annotation. This layered data, categorized into synthetic composites, realistic single-object, and multi-object videos, provides crucial supervision for the model.

Vera employs a Mixture-of-Transformers (MoT) architecture, utilizing three specialized DiTs for distinct outputs: an edit layer, an alpha matte, and a composite layer. This design allows each component to specialize while enabling cross-layer interaction.

Evaluations show Vera significantly outperforms existing baselines in content preservation. Human preference studies with creative reviewers further validated Vera's superiority in maintaining original content and adhering to instructions, with comparable or better video quality.

VOID: Physically Plausible Object Removal

VOID addresses the challenge of removing objects while maintaining physical consistency. Existing methods often fail to account for how an object's removal impacts the scene's dynamics, leading to unnatural results.

VOID utilizes a two-pass pipeline. First, a reasoning pipeline identifies causally affected regions, guiding a diffusion model to generate a physically plausible counterfactual video. A second pass refines the output to prevent artifacts like object morphing.

The training data for VOID is generated using the Kubric simulation engine and HUMOTO motion capture data, creating synthetic counterfactuals that adhere to physical laws. This ensures that when an object is removed, the scene reacts realistically.

Key improvements include quadmask conditioning, which explicitly identifies regions likely to change, and a second-pass refiner for visual stability. VOID is trained on the CogVideoX-Fun-V1.5, 5b-InP backbone, fine-tuned for interaction-aware inpainting.

Experiments demonstrate VOID's superior ability to maintain consistent scene dynamics compared to prior methods. User studies confirm its effectiveness, with participants overwhelmingly selecting VOID's outputs as the most realistic and physically plausible.

These projects represent a significant stride toward more controllable AI video editing, aligning with Netflix's commitment to serving both creators and members. While production-ready quality requires further refinement, Vera and VOID highlight the potential of generative AI for video editing to revolutionize creative workflows.