Transform Video into Audio: Integrating MMAudio Cognitive Actions

23 Apr 2025
Transform Video into Audio: Integrating MMAudio Cognitive Actions

In the world of multimedia applications, the ability to seamlessly convert video content into high-quality audio can significantly enhance user experience. The MMAudio Cognitive Actions provide a powerful solution for developers looking to implement video-to-audio synthesis in their applications. By leveraging pre-built actions, you can save time and effort while delivering contextually rich audio that aligns with visual elements.

Prerequisites

To get started with the MMAudio Cognitive Actions, you'll need to obtain an API key from the Cognitive Actions platform. This key will be used to authenticate your requests. Generally, you would include this API key in the headers of your HTTP requests to securely access the service.

Cognitive Actions Overview

Add Sound to Video

The Add Sound to Video action transforms visual content into high-quality audio using the MMAudio Video-to-Audio Synthesis Model. This action allows you to generate contextually appropriate and temporally synchronized audio from a video file.

Input

The input for this action is structured as follows:

  • seed (integer): A seed for random number generation. Use -1 for a randomized seed.
  • video (string): A URI pointing to the video file for generating audio. The video must be accessible at the provided URI.
  • prompt (string): Textual input to guide the audio generation process with the desired content.
  • duration (number): The duration of the generated audio output in seconds. Default is 8 seconds.
  • numberOfSteps (integer): The number of inference steps during generation. Higher values may yield higher-quality audio.
  • guidanceStrength (number): Controls the CFG strength, impacting adherence to the prompt (0-10).
  • negativeTextPrompt (string): A prompt to specify sounds to avoid during generation.

Example Input:

{
  "seed": -1,
  "video": "https://huggingface.co/hkchengrex/MMAudio/resolve/main/examples/sora_galloping.mp4",
  "prompt": "galloping",
  "duration": 8,
  "numberOfSteps": 25,
  "guidanceStrength": 4.5,
  "negativeTextPrompt": "music"
}

Output

Upon successful execution, the action returns a URI pointing to the generated audio file.

Example Output:

https://assets.cognitiveactions.com/invocations/a3b9c991-5080-417f-9170-79da9af56be5/b3e025e6-916f-4b2c-ab63-fb5f3c9eb9cf.mp4

Conceptual Usage Example (Python)

Here is a conceptual Python code snippet demonstrating how to call the Add Sound to Video action:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute" # Hypothetical endpoint

action_id = "e91da708-74d5-4d45-940d-f540b3201e97" # Action ID for Add Sound to Video

# Construct the input payload based on the action's requirements
payload = {
    "seed": -1,
    "video": "https://huggingface.co/hkchengrex/MMAudio/resolve/main/examples/sora_galloping.mp4",
    "prompt": "galloping",
    "duration": 8,
    "numberOfSteps": 25,
    "guidanceStrength": 4.5,
    "negativeTextPrompt": "music"
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload} # Hypothetical structure
    )
    response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this code snippet, replace YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key. The action ID for Add Sound to Video is provided, and the input payload is structured according to the action's requirements. The endpoint URL and request structure are illustrative and may vary in actual implementation.

Conclusion

The MMAudio Cognitive Actions offer developers a robust method for integrating video-to-audio synthesis into their applications. By utilizing the Add Sound to Video action, you can enrich your multimedia projects with high-quality audio that complements visual content. Explore the possibilities of audio generation and consider how these actions could enhance your next development project!

← Back to All Articles