Vace Outpainting, new Depth extractor

docs/VACE.md

@@ -1,6 +1,6 @@
 # VACE ControlNet Guide
 
-VACE is a powerful ControlNet that enables Video-to-Video and Reference-to-Video generation. It allows you to inject your own images into output videos, animate characters, perform inpainting/outpainting, and continue videos.
+VACE is a powerful ControlNet that enables Video-to-Video and Reference-to-Video generation. It allows you to inject your own images into output videos, animate characters, perform inpainting/outpainting, and continue existing videos.
 
 ## Overview
 
@@ -10,7 +10,8 @@ VACE is probably one of the most powerful Wan models available. With it, you can
 - Perform video inpainting and outpainting
 - Continue existing videos
 - Transfer motion from one video to another
-- Change the style of scenes while preserving depth
+- Change the style of scenes while preserving the structure of the scenes
+
 
 ## Getting Started
 
@@ -18,79 +19,93 @@ VACE is probably one of the most powerful Wan models available. With it, you can
 1. Select either "Vace 1.3B" or "Vace 13B" from the dropdown menu
 2. Note: VACE works best with videos up to 7 seconds with the Riflex option enabled
 
+You can also use any derived Vace models such as Vace Fusionix or combine Vace with Loras accelerator such as Causvid.
+
 ### Input Types
 
-VACE accepts three types of visual hints (which can be combined):
-
 #### 1. Control Video
-- Transfer motion or depth to a new video
-- Use only the first n frames and extrapolate the rest
-- Perform inpainting with grey color (127) as mask areas
-- Grey areas will be filled based on text prompt and reference images
+The Control Video is the source material that contains the instructions about what you want. So Vace expects in the Control Video some visual hints about the type of processing expected: for instance replacing an area by something else, converting an Open Pose wireframe into a human motion, colorizing an Area,  transferring the depth of an image area, ...
+
+For example, anywhere your control video contains the color 127 (grey), it will be considered as an area to be inpainting and replaced by the content of your text prompt and / or a reference image (see below). Likewise if the frames of a Control Video contains an Open Pose wireframe (basically some straight lines tied together that describes the pose of a person), Vace will automatically turn this Open Pose into a real human based on the text prompt and any reference Images (see below).
+
+You can either build yourself the Control Video with the annotators tools provided by the Vace team (see the Vace ressources at the bottom) or you can let WanGP (recommended option) generates on the fly a Vace formatted Control Video based on information you provide.
+
+WanGP wil need the following information to generate a Vace Control Video:
+- A *Control Video* : this video shouldn't have been altered by an annotator tool and can be taken straight from youtube or your camera
+- *Control Video Process* : This is the type of process you want to apply on the control video. For instance *Transfer Human Motion* will generate the Open Pose information from your video so that you can transfer this same motion to a generated character
+- *Area Processed* : you can target the processing to a specific area. For instance even if there are multiple people in the Control Video you may want to replace only one them. If you decide to target an area you will need to provide a *Video Mask* as well. These types of videos can be easily created using the Matanyone tool embedded with WanGP (see the doc of Matanyone below). WanGP can apply different types of process, one the mask and another one on the outside the mask.
+
+Another nice thing is that you can combine all effects above with Outpainting since WanGP will create automatically an outpainting area in the Control Video if you ask for this. 
+
+By default WanGP will ask Vace to generate new frames in the "same spirit" of the control video if the latter is shorter than the number frames that you have requested.
+
 
 #### 2. Reference Images
-- Use as background/setting for the video
-- Inject people or objects of your choice
-- Select multiple reference images
-- **Tip**: Replace complex backgrounds with white for better object integration
-- Always describe injected objects/people explicitly in your text prompt
+With Reference Images you can inject people or objects of your choice in the Video.
+You can also force Images to appear at a specific frame nos in the Video.
 
-#### 3. Video Mask
-- Stronger control over which parts to keep (black) or replace (white)
-- Perfect for inpainting/outpainting
-- Example: White mask except at beginning/end (black) keeps first/last frames while generating middle content
+If the Reference Image is a person or an object, it is recommended to turn on the background remover that will replace the background by the white color.
+This is not needed for a background image or an injected frame at a specific position.
 
-## Common Use Cases
+It is recommended to describe injected objects/people explicitly in your text prompt so that Vace can connect the Reference Images to the new generated video and this will increase the chance that you will find your injected people or objects.
 
-### Motion Transfer
-**Goal**: Animate a character of your choice using motion from another video
-**Setup**: 
-- Reference Images: Your character
-- Control Video: Person performing desired motion
-- Text Prompt: Describe your character and the action
 
-### Object/Person Injection
-**Goal**: Insert people or objects into a scene
-**Setup**:
-- Reference Images: The people/objects to inject
-- Text Prompt: Describe the scene and explicitly mention the injected elements
+### Understanding Vace Control Video and Mask format
+As stated above WanGP will adapt the Control Video and the Video Mask to meet your instructions. You can preview the first frames of the new Control Video and of the Video Mask in the Generation Preview box (just click a thumbnail) to check that your request has been properly interpreted. You can as well ask WanGP to save in the main folder of WanGP the full generated Control Video and  Video Mask by launching the app with the *--save-masks* command.
 
-### Character Animation
-**Goal**: Animate a character based on text description
-**Setup**:
-- Control Video: Video of person moving
-- Text Prompt: Detailed description of your character
+Look at the background colors of both the Control Video and the Video Mask:
+The Mask Video is the most important because depending on the color of its pixels, the Control Video will be interpreted differently. If an area in the Mask is black, the corresponding Control Video area will be kept as is. On the contrary if an area of the Mask is plain white, a Vace process will be applied on this area. If there isn't any Mask Video the Vace process will apply on the whole video frames. The nature of the process itself will depend on what there is in the Control Video for this area. 
+- if the area in grey (127) in the Control Video, this area will be replaced by new content based on the text prompt or image references
+- if an area represents a person in the wireframe Open Pose format, it will be replaced by a person animated with motion described by the Open Pose.The appearance of the person will depend on the text prompt or image references
+- if an area contains multiples shades of grey, these will be assumed to represent different levels of image depth and Vace will try to generate new content located at the same depth
 
-### Style Transfer with Depth
-**Goal**: Change scene style while preserving spatial relationships
-**Setup**:
-- Control Video: Original video (for depth information)
-- Text Prompt: New style description
+There are more Vace representations. For all the different mapping please refer the official Vace documentation.
 
-## Integrated Matanyone Tool
+### Example 1 : Replace a Person in one video by another one by keeping the Background
+1) In Vace, select *Control Video Process*=**Transfer human pose**, *Area processed*=**Masked area** 
+2) In *Matanyone Video Mask Creator*, load your source video and create a mask where you targetted a specific person 
+3) Click *Export to Control Video Input and Video Mask Input* to transfer both the original video that now becomes the *Control Video* and the black & white mask that now defines the *Video Mask Area*
+4) Back in Vace, in *Reference Image* select **Inject Landscapes / People / Objects** and upload one or several pictures of the new person
+5) Generate
 
-WanGP includes the Matanyone tool, specifically tuned for VACE workflows. This helps create control videos and masks simultaneously.
+This works also with several people at the same time (you just need to mask several people in *Matanyone*), you can also play with the slider *Expand / Shrink Mask* if the new person is larger than the original one and of course, you can also use the text *Prompt* if you dont want to use an image for the swap.
 
-### Creating Face Replacement Masks
+
+### Example 2 : Change the Background behind some characters
+1) In Vace, select *Control Video Process*=**Inpainting**, *Area processed*=**Non Masked area** 
+2) In *Matanyone Video Mask Creator*, load your source video and create a mask where you targetted the people you want to keep 
+3) Click *Export to Control Video Input and Video Mask Input* to transfer both the original video that now becomes the *Control Video* and the black & white mask that now defines the *Video Mask Area*
+4) Generate
+
+If instead *Control Video Process*=**Depth**, then the background although it will be still different it will have a similar geometry than in the control video
+
+### Example 3 : Outpaint a Video to the Left and Inject a Character in this new area 
+1) In Vace, select *Control Video Process*=**Keep Unchanged** 
+2) *Control Video Outpainting in Percentage* enter the value 40 to the *Left* entry
+3) In *Reference Image* select **Inject Landscapes / People / Objects** and upload one or several pictures of a person
+4) Enter the *Prompt* such as "a person is coming from the left" (you will need of course a more accurate description)
+5) Generate
+
+
+
+### Creating Face / Object Replacement Masks
 1. Load your video in Matanyone
-2. Click on the face in the first frame
-3. Create a mask for the face
-4. Generate both control video and mask video with "Generate Video Matting"
-5. Export to VACE with "Export to current Video Input and Video Mask"
-6. Load replacement face image in Reference Images field
+2. Click on the face or object in the first frame
+3. Validate the mask by clicking **Set Mask**
+4. Generate a copy of the control video (for easy transfers) and a new mask video by clicking "Generate Video Matting"
+5. Export to VACE with *Export to Control Video Input and Video Mask Input*
 
 ### Advanced Matanyone Tips
-- **Negative Point Prompts**: Remove parts from current selection
-- **Sub Masks**: Create multiple independent masks, then combine them
-- **Background Masks**: Select everything except the character (useful for background replacement)
-- Enable/disable sub masks in Matanyone settings
+- **Negative Point Prompts**: Remove parts from current selection if the mask goes beyond the desired area
+- **Sub Masks**: Create multiple independent masks, then combine them. This may be useful if you are struggling to select exactly what you want.    
+
 
 ## Recommended Settings
 
 ### Quality Settings
-- **Skip Layer Guidance**: Turn ON with default configuration for better results
+- **Skip Layer Guidance**: Turn ON with default configuration for better results (useless with FusioniX of Causvid are there is no cfg)
 - **Long Prompts**: Use detailed descriptions, especially for background elements not in reference images
-- **Steps**: Use at least 15 steps for good quality, 30+ for best results
+- **Steps**: Use at least 15 steps for good quality, 30+ for best results if you use the original Vace model. But only 8-10 steps are sufficient with Vace Funsionix or if you use Loras such as Causvid or Self-Forcing.
 
 ### Sliding Window Settings
 For very long videos, configure sliding windows properly:
@@ -98,27 +113,24 @@ For very long videos, configure sliding windows properly:
 - **Window Size**: Set appropriate duration for your content
 - **Overlap Frames**: Long enough for motion continuity, short enough to avoid blur propagation
 - **Discard Last Frames**: Remove at least 4 frames from each window (VACE 1.3B tends to blur final frames)
+- **Add Overlapped Noise**: May or may not reduce quality degradation over time
 
 ### Background Removal
-VACE includes automatic background removal options:
+WanGP includes automatic background removal options:
 - Use for reference images containing people/objects
-- **Don't use** for landscape/setting reference images (first reference image)
-- Multiple background removal types available
+- **Don't use** this for landscape/setting reference images (the first reference image)
+- If you are not happy with the automatic background removal tool you can use the Image version of Matanyone for a precise background removal
 
 ## Window Sliding for Long Videos
 
 Generate videos up to 1 minute by merging multiple windows:
+The longer the video the greater the quality degradation. However the effect will be less visible if your generated video reuses mostly non altered control video.
 
 ### How It Works
 - Each window uses corresponding time segment from control video
 - Example: 0-4s control video → first window, 4-8s → second window, etc.
 - Automatic overlap management ensures smooth transitions
 
-### Settings
-- **Window Size**: Duration of each generation window
-- **Overlap Frames**: Frames shared between windows for continuity
-- **Discard Last Frames**: Remove poor-quality ending frames
-- **Add Overlapped Noise**: Reduce quality degradation over time
 
 ### Formula
 ```
@@ -133,10 +145,7 @@ Generated Frames = [Windows - 1] × [Window Size - Overlap - Discard] + Window S
 ## Advanced Features
 
 ### Extend Video
-Click "Extend the Video Sample, Please!" during generation to add more windows dynamically.
-
-### Noise Addition
-Add noise to overlapped frames to hide accumulated errors and quality degradation.
+Click "Extend the Video Sample, Please!" during generation to add more windows dynamically. This can be useful, if you don't ask for many frames to generated initially and are happy the video parts already generated.
 
 ### Frame Truncation
 Automatically remove lower-quality final frames from each window (recommended: 4 frames for VACE 1.3B).
@@ -181,7 +190,6 @@ Automatically remove lower-quality final frames from each window (recommended: 4
 4. Check control video quality
 
 ## Tips for Best Results
-
 1. **Detailed Prompts**: Describe everything in the scene, especially elements not in reference images
 2. **Quality Reference Images**: Use high-resolution, well-lit reference images
 3. **Proper Masking**: Take time to create precise masks with Matanyone