Harnessing Image Generation with pnyompen/sdxl-controlnet-lora-small Cognitive Actions

24 Apr 2025
Harnessing Image Generation with pnyompen/sdxl-controlnet-lora-small Cognitive Actions

In the world of AI-driven creativity, the pnyompen/sdxl-controlnet-lora-small offers a powerful suite of Cognitive Actions aimed specifically at image generation. By leveraging the SDXL Canny ControlNet along with LoRA support, developers can seamlessly generate images from text prompts, customize various parameters, and enhance the artistic output with precision. This article will walk you through the process of integrating these actions into your applications, exploring their capabilities and providing useful examples.

Prerequisites

To begin using the Cognitive Actions associated with the pnyompen/sdxl-controlnet-lora-small, you'll need to ensure you have the following:

  1. An API key for the Cognitive Actions platform.
  2. Basic familiarity with making HTTP requests and handling JSON payloads.

Conceptually, authentication typically involves passing your API key in the headers of your requests, allowing you to access the Cognitive Actions securely.

Cognitive Actions Overview

Generate Image with SDXL Canny ControlNet and LoRA

This action utilizes SDXL Canny ControlNet with LoRA support to create images based on text prompts. You can adjust parameters like denoising strength, LoRA scaling, and classifier-free guidance to refine the image generation process.

  • Category: Image Generation

Input

The input schema for this action is defined as follows:

{
  "seed": 1234,
  "image": "https://example.com/input-image.png",
  "prompt": "A serene landscape at sunset",
  "img2img": false,
  "clipSkip": 1,
  "strength": 0.8,
  "loraScale": 0.95,
  "scheduler": "K_EULER",
  "loraWeights": "https://example.com/lora-weights.tar",
  "guidanceScale": 7.5,
  "conditionScale": 1.1,
  "ipAdapterScale": 1,
  "negativePrompt": "A crowded city",
  "numberOfOutputs": 1,
  "autoGenerateCaption": false,
  "generatedCaptionWeight": 0.5,
  "numberOfInferenceSteps": 30
}

The following fields are available in the input schema:

  • image: (string) URL of the input image (used in img2img mode).
  • prompt: (string) Text prompt guiding the image generation.
  • img2img: (boolean) Whether to use img2img pipeline.
  • strength: (number) Denoising strength for img2img.
  • loraScale: (number) Adjusts LoRA additive scale.
  • scheduler: (string) Scheduler to use for the generation process.
  • numberOfOutputs: (integer) Number of images to generate (1 to 4).
  • Other optional fields include negativePrompt, conditionScale, and autoGenerateCaption.

Example Input

Here's an example of a JSON payload you might send to this action:

{
  "image": "https://replicate.delivery/pbxt/JiOTMCHj4oGrTTf8Pg2r7vyI8YdXc5jL2IDyC2SfhuggjYe6/out-0%20%281%29.png",
  "prompt": "shot in the style of sksfer, a woman in alaska",
  "img2img": false,
  "strength": 0.8,
  "loraScale": 0.95,
  "scheduler": "K_EULER",
  "loraWeights": "https://pbxt.replicate.delivery/mwN3AFyYZyouOB03Uhw8ubKW9rpqMgdtL9zYV9GF2WGDiwbE/trained_model.tar",
  "guidanceScale": 7.5,
  "conditionScale": 0.5,
  "negativePrompt": "",
  "numberOfOutputs": 1,
  "autoGenerateCaption": false,
  "numberOfInferenceSteps": 40
}

Output

The action typically returns a list of URLs pointing to the generated images. For example:

[
  "https://assets.cognitiveactions.com/invocations/5ef9c2ac-23b0-47ba-a72a-01cc187a1a0e/2184947e-9a0c-4ac6-b22b-fe6c41741e91.webp"
]

This output provides direct access to the generated image(s) based on your input parameters.

Conceptual Usage Example (Python)

Below is a conceptual Python code snippet demonstrating how to call this Cognitive Action using the specified input structure:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute" # Hypothetical endpoint

action_id = "e60d588e-c42d-4fbc-bed2-a84492eff8f1" # Action ID for Generate Image with SDXL Canny ControlNet and LoRA

# Construct the input payload based on the action's requirements
payload = {
    "image": "https://replicate.delivery/pbxt/JiOTMCHj4oGrTTf8Pg2r7vyI8YdXc5jL2IDyC2SfhuggjYe6/out-0%20%281%29.png",
    "prompt": "shot in the style of sksfer, a woman in alaska",
    "img2img": false,
    "strength": 0.8,
    "loraScale": 0.95,
    "scheduler": "K_EULER",
    "loraWeights": "https://pbxt.replicate.delivery/mwN3AFyYZyouOB03Uhw8ubKW9rpqMgdtL9zYV9GF2WGDiwbE/trained_model.tar",
    "guidanceScale": 7.5,
    "conditionScale": 0.5,
    "negativePrompt": "",
    "numberOfOutputs": 1,
    "autoGenerateCaption": false,
    "numberOfInferenceSteps": 40
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload} # Hypothetical structure
    )
    response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this code, you replace the placeholder for the API key and invoke the image generation action by sending the correctly structured payload. The result will include the generated image URLs.

Conclusion

The pnyompen/sdxl-controlnet-lora-small Cognitive Actions provide a robust and versatile approach to image generation, allowing developers to create stunning visuals from simple text prompts. By adjusting various parameters, you can fine-tune the output to meet specific artistic needs. As you explore these capabilities, consider experimenting with different prompts and settings to unlock the full potential of your creativity!

← Back to All Articles