Enhance Image Generation with the SDXL Multi-ControlNet LoRA Actions

23 Apr 2025
Enhance Image Generation with the SDXL Multi-ControlNet LoRA Actions

In today's rapidly advancing digital landscape, image generation has become a critical capability for many applications. The fofr/sdxl-lcm-multi-controlnet-lora provides developers with powerful Cognitive Actions designed specifically for advanced image generation tasks. Utilizing the SDXL LCM model with multi-ControlNet and LoRA capabilities, these actions support various functionalities such as img2img, inpainting, and fine-grained control over multiple conditioning parameters. This article will guide you through integrating these Cognitive Actions into your applications, empowering you to create stunning visuals with ease.

Prerequisites

To get started with the Cognitive Actions in the fofr/sdxl-lcm-multi-controlnet-lora spec, you will need:

  • An API key for the Cognitive Actions platform.
  • Basic knowledge of JSON and RESTful API calls.
  • Familiarity with Python for making API requests (though concepts can be applied in other programming languages).

Authentication typically involves passing your API key in the headers of your requests, allowing secure access to the Cognitive Actions.

Cognitive Actions Overview

Generate with Multi-ControlNet and LoRA

Purpose:
This action utilizes the SDXL LCM model, employing multi-ControlNet and LoRA capabilities for advanced image generation. It supports various modes like img2img and inpainting, enabling developers to create high-quality images through precise conditioning controls.

Category: Image Generation

Input: The action requires a JSON payload structured according to the following schema:

{
  "mask": "uri", // optional
  "seed": "integer", // optional
  "image": "uri", // optional
  "width": "integer", // default: 768
  "height": "integer", // default: 768
  "prompt": "string", // default: "An astronaut riding a rainbow unicorn"
  "loraScale": "number", // default: 0.6
  "controlnet1": "string", // default: "none"
  "controlnet2": "string", // default: "none"
  "controlnet3": "string", // default: "none"
  "refineSteps": "integer", // optional
  "guidanceScale": "number", // default: 1.1
  "applyWatermark": "boolean", // default: true
  "controlnet1End": "number", // default: 1
  "controlnet2End": "number", // default: 1
  "controlnet3End": "number", // default: 1
  "libraryWeights": "string", // optional
  "negativePrompt": "string", // default: ""
  "promptStrength": "number", // default: 0.8
  "numberOfOutputs": "integer", // default: 1
  "refinementStyle": "string", // default: "no_refiner"
  "controlnet1Image": "uri", // optional
  "controlnet1Start": "number", // default: 0
  "controlnet2Start": "number", // default: 0
  "controlnet3Start": "number", // default: 0
  "imageSizingStrategy": "string", // default: "width_height"
  "disableSafetyChecker": "boolean", // default: false
  "numberOfInferenceSteps": "integer", // default: 4
  "controlnet1ConditioningScale": "number", // default: 0.75
  "controlnet2ConditioningScale": "number", // default: 0.75
  "controlnet3ConditioningScale": "number" // default: 0.75
}

Example Input: Here’s an example of a valid JSON input payload for this action:

{
  "width": 1024,
  "height": 1024,
  "prompt": "A TOK photo, extreme macro photo of a golden astronaut riding a unicorn statue, in a museum, 50mm",
  "loraScale": 0.8,
  "controlnet1": "soft_edge_hed",
  "controlnet2": "none",
  "controlnet3": "none",
  "guidanceScale": 1.1,
  "applyWatermark": false,
  "controlnet1End": 0.5,
  "controlnet2End": 1,
  "controlnet3End": 1,
  "libraryWeights": "https://replicate.delivery/pbxt/hKhpVe6O7EwXNCiWORev3OEDRCoWeMlqZMLQDEvwDyHV3hvjA/trained_model.tar",
  "negativePrompt": "rainbow, soft, blurry",
  "promptStrength": 0.9,
  "numberOfOutputs": 1,
  "refinementStyle": "no_refiner",
  "controlnet1Image": "https://replicate.delivery/pbxt/JuVFxyGs9YTy2ce3q8rSCrgYJKJNJdHKxpZUvbdlDTKSExpg/out-0-44.png",
  "controlnet1Start": 0,
  "controlnet2Start": 0,
  "controlnet3Start": 0,
  "imageSizingStrategy": "width_height",
  "numberOfInferenceSteps": 8,
  "controlnet1ConditioningScale": 0.6,
  "controlnet2ConditioningScale": 0.75,
  "controlnet3ConditioningScale": 0.75
}

Output: The action typically returns an array of URLs pointing to the generated images. For example:

[
  "https://assets.cognitiveactions.com/invocations/453de859-95d1-49ff-815a-6c838e7b6e21/538f7498-9aa1-4cf4-a575-5dabbd956069.png",
  "https://assets.cognitiveactions.com/invocations/453de859-95d1-49ff-815a-6c838e7b6e21/21c6562d-6055-468f-81e3-f30a5188d317.png"
]

Conceptual Usage Example (Python): Below is a conceptual Python code snippet demonstrating how to invoke the Generate with Multi-ControlNet and LoRA action:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute" # Hypothetical endpoint

action_id = "810c1305-5d31-41a7-a8ee-32c5cfe8174c" # Action ID for Generate with Multi-ControlNet and LoRA

# Construct the input payload based on the action's requirements
payload = {
    "width": 1024,
    "height": 1024,
    "prompt": "A TOK photo, extreme macro photo of a golden astronaut riding a unicorn statue, in a museum, 50mm",
    "loraScale": 0.8,
    "controlnet1": "soft_edge_hed",
    "controlnet2": "none",
    "controlnet3": "none",
    "guidanceScale": 1.1,
    "applyWatermark": False,
    "controlnet1End": 0.5,
    "controlnet2End": 1,
    "controlnet3End": 1,
    "libraryWeights": "https://replicate.delivery/pbxt/hKhpVe6O7EwXNCiWORev3OEDRCoWeMlqZMLQDEvwDyHV3hvjA/trained_model.tar",
    "negativePrompt": "rainbow, soft, blurry",
    "promptStrength": 0.9,
    "numberOfOutputs": 1,
    "refinementStyle": "no_refiner",
    "controlnet1Image": "https://replicate.delivery/pbxt/JuVFxyGs9YTy2ce3q8rSCrgYJKJNJdHKxpZUvbdlDTKSExpg/out-0-44.png",
    "controlnet1Start": 0,
    "controlnet2Start": 0,
    "controlnet3Start": 0,
    "imageSizingStrategy": "width_height",
    "numberOfInferenceSteps": 8,
    "controlnet1ConditioningScale": 0.6,
    "controlnet2ConditioningScale": 0.75,
    "controlnet3ConditioningScale": 0.75
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload} # Hypothetical structure
    )
    response.raise_for_status()  # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this snippet, replace YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key. The payload variable is structured according to the action's requirements, and the request sends the action ID and inputs to the hypothetical endpoint.

Conclusion

The fofr/sdxl-lcm-multi-controlnet-lora spec offers a powerful suite of Cognitive Actions for developers looking to leverage advanced image generation capabilities. By utilizing the flexibility of LoRA and ControlNet, you can create high-quality images tailored to your specific needs. Whether you're developing applications for art, design, or any other domain, integrating these actions can significantly enhance your workflow. Start experimenting with the actions today and unlock new creative possibilities!

← Back to All Articles