Create Stunning Visuals with Audio: A Guide to the Stable Diffusion Dance Cognitive Actions

22 Apr 2025
Create Stunning Visuals with Audio: A Guide to the Stable Diffusion Dance Cognitive Actions

Integrating audio and visual elements has never been easier than with the Stable Diffusion Dance Cognitive Actions. This powerful API allows developers to generate a sequence of images influenced by audio inputs and textual prompts using the advanced Stable Diffusion model. With these pre-built actions, you can create immersive experiences that blend sound with visuals, opening up endless possibilities for creative applications, from music videos to dynamic presentations.

Prerequisites

Before getting started with the Stable Diffusion Dance Cognitive Actions, ensure you have the following:

  • An API key for the Cognitive Actions platform.
  • Familiarity with JSON structure, as you will be working with JSON payloads for input and output.
  • Basic knowledge of making HTTP requests in your preferred programming language.

Authentication typically involves passing your API key in the request headers to securely access the Cognitive Actions service.

Cognitive Actions Overview

Generate Audio-Influenced Image Sequence

The Generate Audio-Influenced Image Sequence action creates a series of images based on audio input and textual prompts. This action is part of the video-to-audio-synthesis category and allows customization of various parameters like image dimensions, style, and the degree of audio influence.

Input

The input for this action requires a structured JSON payload containing several fields:

  • width (integer, default: 384): Specifies the width of the generated image. Recommended is 512 for optimal results.
  • height (integer, default: 512): Specifies the height of the generated image. Again, 512 is ideal for coherence.
  • prompts (string): A string of prompts separated by newline characters that guide the image generation.
  • audioFile (string, URI): URL to the audio file that will influence the image creation.
  • batchSize (integer, default: 24): Number of images to generate at once.
  • frameRate (number, default: 16): Frames per second for the generated sequence.
  • randomSeed (integer, default: 13): Seed for random number generation, influencing the variability of outputs.
  • promptScale (number, default: 15): Degree of prompt influence on the generated image.
  • styleSuffix (string, default: "by paul klee, intricate details"): Consistent style applied across images.
  • audioSmoothing (number, default: 0.8): Factor for smoothing audio input.
  • diffusionSteps (integer, default: 20): Number of steps for diffusion, affecting image quality and generation time.
  • audioNoiseScale (number, default: 0.3): Extent to which audio affects the images.
  • audioLoudnessType (string, enum: "rms", "peak", default: "peak"): Type of loudness for audio processing.
  • frameInterpolation (boolean, default: true): Whether to apply frame interpolation.

Example Input:

{
  "width": 512,
  "height": 512,
  "prompts": "A painting of a moth\nA painting of a killer dragonfly by paul klee, intricate detail\nTwo fishes talking to eachother in deep sea, art by hieronymus bosch",
  "audioFile": "https://replicate.delivery/pbxt/MjYAIYsg89ldJjFQtKgai1Tpl9urAqeCrx2PHJCemHO8r6iY/test1.mp3",
  "batchSize": 24,
  "frameRate": 16,
  "randomSeed": 13,
  "promptScale": 15,
  "styleSuffix": "by paul klee, intricate details",
  "audioSmoothing": 0.8,
  "diffusionSteps": 20,
  "audioNoiseScale": 0.3,
  "audioLoudnessType": "peak",
  "frameInterpolation": true
}

Output

The action returns an array of URLs pointing to the generated images and videos. The output includes both still images and video files created from the audio and prompts provided.

Example Output:

[
  "https://assets.cognitiveactions.com/invocations/5db1e2cb-cf69-4a32-8a97-55944605aaa4/773fd2f6-2bcb-43b8-be74-2ae53eb19e19.png",
  "https://assets.cognitiveactions.com/invocations/5db1e2cb-cf69-4a32-8a97-55944605aaa4/bc446bf5-d150-4e73-af25-97d6ffbb41a9.png",
  "https://assets.cognitiveactions.com/invocations/5db1e2cb-cf69-4a32-8a97-55944605aaa4/ab527f10-1f5c-4122-9888-2c60b7738233.png",
  "https://assets.cognitiveactions.com/invocations/5db1e2cb-cf69-4a32-8a97-55944605aaa4/3e630581-5e34-42da-ba24-c2c3b74e6fe6.png",
  "https://assets.cognitiveactions.com/invocations/5db1e2cb-cf69-4a32-8a97-55944605aaa4/c33f05fc-ab98-41a4-9ed7-1b1ae79826a6.mp4",
  "https://assets.cognitiveactions.com/invocations/5db1e2cb-cf69-4a32-8a97-55944605aaa4/1a0fa26f-0173-454d-9dad-f95aeb5320e3.mp4"
]

Conceptual Usage Example (Python)

Here's a conceptual Python code snippet demonstrating how to call the Cognitive Actions execution endpoint for the Generate Audio-Influenced Image Sequence action:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute"  # Hypothetical endpoint

action_id = "8ecf6190-5f15-4109-8551-b28f12ec3d38"  # Action ID for Generate Audio-Influenced Image Sequence

# Construct the input payload based on the action's requirements
payload = {
    "width": 512,
    "height": 512,
    "prompts": "A painting of a moth\nA painting of a killer dragonfly by paul klee, intricate detail\nTwo fishes talking to eachother in deep sea, art by hieronymus bosch",
    "audioFile": "https://replicate.delivery/pbxt/MjYAIYsg89ldJjFQtKgai1Tpl9urAqeCrx2PHJCemHO8r6iY/test1.mp3",
    "batchSize": 24,
    "frameRate": 16,
    "randomSeed": 13,
    "promptScale": 15,
    "styleSuffix": "by paul klee, intricate details",
    "audioSmoothing": 0.8,
    "diffusionSteps": 20,
    "audioNoiseScale": 0.3,
    "audioLoudnessType": "peak",
    "frameInterpolation": true
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload}  # Hypothetical structure
    )
    response.raise_for_status()  # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this code snippet, replace YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key. The payload variable includes the input structure required for the action. The hypothetical endpoint and request structure are designed to help you understand how to interact with the Cognitive Actions API effectively.

Conclusion

The Stable Diffusion Dance Cognitive Actions provide an innovative way to merge audio and visual content, enabling developers to create captivating experiences. By leveraging the powerful capabilities of the Stable Diffusion model, you can generate stunning image sequences influenced by sound, enhancing your applications in unique ways.

Now that you understand how to use these Cognitive Actions, consider experimenting with different audio files and prompts to see how they influence your image generation. Happy coding!

← Back to All Articles