Unlocking Multimodal Insights with lucataco/smolvlm-instruct Cognitive Actions

23 Apr 2025
Unlocking Multimodal Insights with lucataco/smolvlm-instruct Cognitive Actions

The lucataco/smolvlm-instruct API offers powerful Cognitive Actions for developers focusing on multimodal analysis, enabling efficient integration of image and text processing capabilities into applications. The standout action, Perform Multimodal Analysis, allows you to leverage the SmolVLM model for tasks like image captioning, visual question answering, and storytelling based on visual content. By utilizing these pre-built actions, developers can enhance user experiences and derive deeper insights from visual data effortlessly.

Prerequisites

Before getting started with the Cognitive Actions, ensure you have the following:

  • An API key for the Cognitive Actions platform. This key is essential for authenticating your API requests.
  • Familiarity with making HTTP requests, as you will need to send JSON payloads to the API endpoint.

Authentication typically involves including your API key in the headers of your requests, ensuring that your application can securely access the Cognitive Actions services.

Cognitive Actions Overview

Perform Multimodal Analysis

The Perform Multimodal Analysis action utilizes the SmolVLM model to generate descriptive text outputs from image and text inputs. This action is particularly useful for applications that require understanding and interpreting visual data, such as generating captions for images or answering questions based on visual content.

Input

The required input schema for this action includes:

  • image (string, required): The URI of the input image to process. The image should be accessible via the provided URL.
  • prompt (string, optional): A text prompt guiding the model's response. It provides context or questions related to the image. The default prompt asks for a description of the image.
  • maxNewTokens (integer, optional): Specifies the maximum number of tokens to generate in the model's response, with acceptable values ranging from 1 to 2000. The default is set to 500.

Example Input:

{
  "image": "https://replicate.delivery/pbxt/M41uQ4M8J9FEqxRJ0tNnliJF2PNJIeGjdid66k2uHOLgv5OJ/weather.png",
  "prompt": "Where do the severe droughts happen according to this image?",
  "maxNewTokens": 500
}

Output

The action typically returns a text output that provides an answer or description based on the image and prompt.

Example Output:

The severe droughts happen in eastern and southern Africa.

Conceptual Usage Example (Python)

Here’s how you might call the Perform Multimodal Analysis action using Python:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute" # Hypothetical endpoint

action_id = "72a1f1ea-6aa0-4496-8a3b-ef20ceffa04a" # Action ID for Perform Multimodal Analysis

# Construct the input payload based on the action's requirements
payload = {
    "image": "https://replicate.delivery/pbxt/M41uQ4M8J9FEqxRJ0tNnliJF2PNJIeGjdid66k2uHOLgv5OJ/weather.png",
    "prompt": "Where do the severe droughts happen according to this image?",
    "maxNewTokens": 500
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload} # Hypothetical structure
    )
    response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this code snippet, replace YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key. The action ID for Perform Multimodal Analysis is set accordingly. The payload is structured based on the required input schema, which includes the image URL and prompt. The response from the API is then processed and printed.

Conclusion

The lucataco/smolvlm-instruct Cognitive Actions provide developers with robust tools for multimodal analysis, enhancing applications with advanced capabilities for interpreting and generating insights from visual data. By integrating these actions, you can create more engaging and informative user experiences. Dive deeper into multimodal applications and explore various use cases to leverage the power of visual understanding in your projects!

← Back to All Articles