Enhance Your Videos with ProPainter: A Guide to Video Inpainting

22 Apr 2025
Enhance Your Videos with ProPainter: A Guide to Video Inpainting

In the world of video editing and processing, the need for sophisticated tools to enhance and manipulate content is paramount. The ProPainter Cognitive Actions provide developers with powerful capabilities to perform advanced video inpainting tasks. This API integration allows users to remove objects, complete masked videos, and expand video views using state-of-the-art flow-based propagation and spatiotemporal Transformers. By leveraging these pre-built actions, developers can save time, improve video quality, and create stunning visual experiences.

Prerequisites

Before diving into the integration of ProPainter Cognitive Actions, ensure you have the following:

  • An API key for the Cognitive Actions platform.
  • Basic familiarity with JSON and HTTP requests.
  • A working environment for executing Python code.

Authentication typically involves passing your API key in the request headers, ensuring secure access to the service.

Cognitive Actions Overview

Perform Video Inpainting with ProPainter

The Perform Video Inpainting with ProPainter action is designed to manipulate video content by removing unwanted objects and filling in gaps seamlessly. This action uses advanced algorithms to ensure that the output video retains a natural look and feel.

  • Category: Video Processing

Input

The input for this action requires several fields, with the video field being mandatory. Below is the schema and an example of the expected input:

{
  "mask": "https://replicate.delivery/pbxt/JgGNe7IjsavH2is00AVgbEwWroEtseid6f3HZIHI7MJH9Num/bmx-trees-mask4.mp4",
  "video": "https://replicate.delivery/pbxt/JgGNdl0r9V14YxgJ4JlsCz71rSU9y8NJD9RDDdJo2Ua3r8C7/bmx-trees2.mp4",
  "width": -1,
  "height": -1,
  "scaleWidth": 1,
  "resizeScale": 1,
  "scaleHeight": 1,
  "maskDilation": 4,
  "processingMode": "video_inpainting",
  "raftIterations": 20,
  "referenceStride": 10,
  "subvideoDuration": 80,
  "useHalfPrecision": true,
  "includeInputVideo": true,
  "saveFramesPerSecond": 24,
  "neighboringFramesLength": 10
}
  • Required Fields:
    • video: URI of the input video (required).
  • Optional Fields:
    • mask: URI of the mask for video inpainting.
    • width and height: Dimensions for processing (default -1 allows auto-determination).
    • processingMode: Defines the operation mode (video_inpainting or video_outpainting).
    • Other fields control scaling, mask dilation, and performance options.

Output

The action typically returns an array of URIs pointing to the processed video outputs. Here’s an example of what the output might look like:

[
  "https://assets.cognitiveactions.com/invocations/74f8d47e-9d96-4ff6-83c4-2c7e89dcf7b7/4144277f-a9cc-4664-a7b4-b49bdd46f3dd.mp4",
  "https://assets.cognitiveactions.com/invocations/74f8d47e-9d96-4ff6-83c4-2c7e89dcf7b7/921b1389-9fa9-4c77-b94a-bfcca2237269.mp4"
]

This output provides direct links to the processed video files.

Conceptual Usage Example (Python)

Below is a conceptual Python code snippet demonstrating how to invoke the Perform Video Inpainting with ProPainter action:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute"  # Hypothetical endpoint

action_id = "d85a5753-7472-4b8a-978c-ec7d865eb96e"  # Action ID for Perform Video Inpainting

# Construct the input payload based on the action's requirements
payload = {
    "mask": "https://replicate.delivery/pbxt/JgGNe7IjsavH2is00AVgbEwWroEtseid6f3HZIHI7MJH9Num/bmx-trees-mask4.mp4",
    "video": "https://replicate.delivery/pbxt/JgGNdl0r9V14YxgJ4JlsCz71rSU9y8NJD9RDDdJo2Ua3r8C7/bmx-trees2.mp4",
    "width": -1,
    "height": -1,
    "scaleWidth": 1,
    "resizeScale": 1,
    "scaleHeight": 1,
    "maskDilation": 4,
    "processingMode": "video_inpainting",
    "raftIterations": 20,
    "referenceStride": 10,
    "subvideoDuration": 80,
    "useHalfPrecision": True,
    "includeInputVideo": True,
    "saveFramesPerSecond": 24,
    "neighboringFramesLength": 10
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload}  # Hypothetical structure
    )
    response.raise_for_status()  # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this example, replace the placeholders with your actual API key and endpoint. The payload is structured based on the action's requirements, including the required video and mask URIs.

Conclusion

The ProPainter Cognitive Actions offer remarkable capabilities for video manipulation, enabling developers to create visually stunning content with ease. By integrating the Perform Video Inpainting action, you can enhance your applications with advanced video editing features that serve various use cases, from content creation to automated video editing. Start experimenting with these actions today and elevate the quality of your video projects!

← Back to All Articles