Harnessing Image Generation with pnyompen/sdxl-controlnet-lora-small Cognitive Actions

21 Apr 2025
Harnessing Image Generation with pnyompen/sdxl-controlnet-lora-small Cognitive Actions

In the realm of image generation, the pnyompen/sdxl-controlnet-lora-small spec offers a powerful toolset for developers looking to create stunning visuals from textual descriptions. This set of Cognitive Actions takes advantage of SDXL Canny controlnet with LoRA support, enabling users to generate images from text prompts and perform img2img conversions with enhanced detailing. With these pre-built actions, developers can streamline their applications, saving time and effort while achieving high-quality results.

Prerequisites

Before diving into the Cognitive Actions, ensure that you have the following:

  • An API key for the Cognitive Actions platform, which will be required for authentication.
  • Basic familiarity with HTTP requests and JSON formatting.

Authentication typically involves passing your API key in the headers of your requests.

Cognitive Actions Overview

Generate Image with SDXL Canny Controlnet and LoRA

This action utilizes SDXL Canny controlnet with LoRA support to generate images based on text prompts, supporting img2img conversion and enhanced image detailing.

Input

The input for this action follows a structured schema. Here are the required and optional fields:

  • image (string): URL of the input image for img2img or inpaint mode. (Example: https://replicate.delivery/pbxt/JiOTMCHj4oGrTTf8Pg2r7vyI8YdXc5jL2IDyC2SfhuggjYe6/out-0%20%281%29.png)
  • prompt (string): The text prompt guiding the image generation. (Example: shot in the style of sksfer, a woman in alaska)
  • img2img (boolean): Enable img2img pipeline. (Example: false)
  • strength (number): Denoising strength for img2img. (Example: 0.8)
  • loraScale (number): Scale for LoRA weights influence. (Example: 0.95)
  • scheduler (string): Scheduling algorithm used for processing. (Example: K_EULER)
  • loraWeights (string): URL to the LoRA weights file. (Example: https://pbxt.replicate.delivery/mwN3AFyYZyouOB03Uhw8ubKW9rpqMgdtL9zYV9GF2WGDiwbE/trained_model.tar)
  • guidanceScale (number): Strength of classifier-free guidance during generation. (Example: 7.5)
  • conditionScale (number): ControlNet influence on the output. (Example: 0.5)
  • negativePrompt (string): Defines undesirable attributes. (Example: "")
  • numberOfOutputs (integer): Number of images to generate. (Example: 1)
  • autoGenerateCaption (boolean): Automatically generate captions. (Example: false)
  • numberOfInferenceSteps (integer): Total denoising steps during generation. (Example: 40)

Here is a sample JSON payload for this action:

{
  "image": "https://replicate.delivery/pbxt/JiOTMCHj4oGrTTf8Pg2r7vyI8YdXc5jL2IDyC2SfhuggjYe6/out-0%20%281%29.png",
  "prompt": "shot in the style of sksfer, a woman in alaska",
  "img2img": false,
  "strength": 0.8,
  "loraScale": 0.95,
  "scheduler": "K_EULER",
  "loraWeights": "https://pbxt.replicate.delivery/mwN3AFyYZyouOB03Uhw8ubKW9rpqMgdtL9zYV9GF2WGDiwbE/trained_model.tar",
  "guidanceScale": 7.5,
  "conditionScale": 0.5,
  "negativePrompt": "",
  "numberOfOutputs": 1,
  "autoGenerateCaption": false,
  "numberOfInferenceSteps": 40
}

Output

Upon successful execution, the action typically returns an array of URLs to the generated images. Here's an example output:

[
  "https://assets.cognitiveactions.com/invocations/7d123595-9a26-44b5-bfcd-69ce62f624c0/936f72be-6aca-4d5c-8e3a-2341dfb01353.webp"
]

Conceptual Usage Example (Python)

Here’s a conceptual Python code snippet demonstrating how to call this action using a hypothetical endpoint:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute" # Hypothetical endpoint

action_id = "9c6e6f54-7541-46b1-b900-62234002d487" # Action ID for Generate Image with SDXL Canny Controlnet and LoRA

# Construct the input payload based on the action's requirements
payload = {
    "image": "https://replicate.delivery/pbxt/JiOTMCHj4oGrTTf8Pg2r7vyI8YdXc5jL2IDyC2SfhuggjYe6/out-0%20%281%29.png",
    "prompt": "shot in the style of sksfer, a woman in alaska",
    "img2img": False,
    "strength": 0.8,
    "loraScale": 0.95,
    "scheduler": "K_EULER",
    "loraWeights": "https://pbxt.replicate.delivery/mwN3AFyYZyouOB03Uhw8ubKW9rpqMgdtL9zYV9GF2WGDiwbE/trained_model.tar",
    "guidanceScale": 7.5,
    "conditionScale": 0.5,
    "negativePrompt": "",
    "numberOfOutputs": 1,
    "autoGenerateCaption": False,
    "numberOfInferenceSteps": 40
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload} # Hypothetical structure
    )
    response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this code, replace YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key, and invoke the action by sending a structured input payload. The endpoint URL and request structure are illustrative, so ensure you adapt them to your actual implementation.

Conclusion

The pnyompen/sdxl-controlnet-lora-small Cognitive Actions provide a robust framework for image generation, leveraging the power of text prompts and advanced algorithms. By integrating these actions into your applications, you can create vibrant, detailed images tailored to your specifications. Whether you're developing a creative tool or enhancing user experience, these actions offer immense potential to elevate your projects. Consider exploring additional use cases or combining them with other Cognitive Actions for even more dynamic capabilities!

← Back to All Articles