Achieve Perfect Lip Sync in Videos with chenxwh/video-retalking Cognitive Actions

24 Apr 2025
Achieve Perfect Lip Sync in Videos with chenxwh/video-retalking Cognitive Actions

Integrating advanced video processing capabilities into your applications can greatly enhance user experience and engagement. The chenxwh/video-retalking API provides a powerful set of Cognitive Actions that facilitate audio-driven lip synchronization and emotion overlay in real-world talking head videos. By leveraging these pre-built actions, developers can save time and effort while achieving high-quality video editing results.

Prerequisites

Before you start using the Cognitive Actions, ensure you have the following:

  • An API key for the Cognitive Actions platform. This key will be used for authentication when making requests.
  • Basic knowledge of how to make HTTP requests and handle JSON data in your application.

For authentication, you will typically pass the API key in the request headers. This is a common practice to ensure that only authorized users can access the API's capabilities.

Cognitive Actions Overview

Perform Audio-based Lip Synchronization

The Perform Audio-based Lip Synchronization action utilizes the VideoReTalking system to edit talking head videos based on input audio. The process involves canonical expression editing, audio-driven lip synchronization, and enhancement of photo-realism. This action is particularly useful for applications that require creating videos where the lip movements of a speaker match the audio perfectly.

Input

The input for this action requires two fields:

  • face: A URI pointing to the video file featuring a talking head.
  • inputAudio: A URI pointing to the audio file that will be used for synchronization.

Here’s an example of the input payload:

{
  "face": "https://replicate.delivery/pbxt/Jnm95KgYvAQIHlR0tg8rbWHweReTtCYp42Drl7dMNtHXaTNR/3.mp4",
  "inputAudio": "https://replicate.delivery/pbxt/JnkUjVcUPLreS4x7ZXXQuCY7qVcLLDNxOeRAsHRi7qj79xBk/1.wav"
}
  • face: This URI must point to a valid video resource.
  • inputAudio: The audio URI should not contain special symbols in the filename to avoid processing errors.

Output

Upon successful execution, the action returns a URI of the edited video with synchronized lips. Here’s an example of the output:

"https://assets.cognitiveactions.com/invocations/e43aa238-768b-4162-b767-ecb9ac2c4546/ae73c8d0-c05a-43fc-b0ed-1b3abb85d75f.mp4"

This output URL links to the resulting video where the lip movements are synchronized with the provided audio.

Conceptual Usage Example (Python)

Here’s how you might call this action using Python:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute" # Hypothetical endpoint

action_id = "2e2e7aed-7883-441b-85e2-09a5a2feddd2" # Action ID for Perform Audio-based Lip Synchronization

# Construct the input payload based on the action's requirements
payload = {
    "face": "https://replicate.delivery/pbxt/Jnm95KgYvAQIHlR0tg8rbWHweReTtCYp42Drl7dMNtHXaTNR/3.mp4",
    "inputAudio": "https://replicate.delivery/pbxt/JnkUjVcUPLreS4x7ZXXQuCY7qVcLLDNxOeRAsHRi7qj79xBk/1.wav"
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload} # Hypothetical structure
    )
    response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this code snippet, replace YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key. The action ID corresponds to the Perform Audio-based Lip Synchronization action, and the payload contains the necessary input URIs. The request is sent to the hypothetical endpoint for execution, and upon success, the result will display the output URI of the synchronized video.

Conclusion

The chenxwh/video-retalking Cognitive Actions offer developers powerful tools for enhancing video content through audio-based lip synchronization. By integrating these actions into your applications, you can create engaging and high-quality video experiences. Consider exploring additional use cases such as dubbing videos or creating educational content where precise lip sync is essential. Start experimenting today and unleash the potential of your video applications!

← Back to All Articles