Unlock Visual Insights: Integrating Visual Question Answering with Llama 3.2-Vision 90B

24 Apr 2025
Unlock Visual Insights: Integrating Visual Question Answering with Llama 3.2-Vision 90B

In today's digital landscape, leveraging artificial intelligence to enhance user interactions is becoming increasingly vital. The lucataco/ollama-llama3.2-vision-90b spec introduces a powerful Cognitive Action that allows developers to harness the capabilities of the Llama 3.2-Vision 90B model for visual reasoning tasks. This action can perform image recognition, captioning, and answer questions about images, making it a valuable tool for applications that require understanding visual content.

Prerequisites

Before diving into the integration of this Cognitive Action, ensure you have the following:

  • API Key: Sign up for access to the Cognitive Actions platform to obtain your API key.
  • Setup: Familiarity with making HTTP requests in your programming environment, particularly using JSON payloads.
  • Authentication: You will need to include your API key in the headers of your requests to authenticate your access.

Cognitive Actions Overview

Perform Visual Question Answering

The Perform Visual Question Answering action allows you to utilize the Llama 3.2-Vision 90B model to analyze images and respond to textual prompts related to those images. This action is categorized under visual-question-answering, optimized for tasks such as image recognition and generating meaningful responses.

Input

The input schema for this action requires the following fields:

  • image (string, required): A URL pointing to the image you want the model to analyze.
  • prompt (string, required): The question or request for information regarding the image.
  • temperature (number, optional): Controls the randomness of the output (default is 0.7).
  • maximumTokens (integer, optional): Sets the maximum number of tokens for the output (default is 512).
  • topProbability (number, optional): Regulates output diversity (default is 0.95).

Example Input:

{
  "image": "https://replicate.delivery/pbxt/M9rZEPYihLWYsHIHMw32oStoi20o9AShqpGzy7WRwz4rHQpp/rococo.jpg",
  "prompt": "Which era does this piece belong to? Give details about the era.",
  "temperature": 0.7,
  "maximumTokens": 512,
  "topProbability": 0.95
}

Output

The action returns an array of strings containing the model's response. The output may vary in structure, but typically consists of detailed information based on the input image and prompt.

Example Output:

[
  "The artwork in question is a ceiling painting, specifically a fresco, which is characteristic of Baroque art. The Baroque period spanned from approximately 1600 to 1750 and was marked by dramatic lighting, intense emotions, and highly ornamented decoration.",
  "**Characteristics of the Artwork:**",
  "* **Dramatic Lighting:** The use of strong contrasts between light and dark creates a sense of drama and tension.",
  "* **Intense Emotions:** The figures depicted in the painting convey powerful emotions through their facial expressions and body language.",
  "* **Ornate Decoration:** Intricate details, such as gilded frames and ornamental patterns, are characteristic of Baroque art.",
  // Additional details...
]

Conceptual Usage Example (Python)

Here's how you might conceptualize calling this Cognitive Action using Python:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute" # Hypothetical endpoint

action_id = "9295127d-7537-4a41-aa98-e0cdd054eaf3"  # Action ID for Perform Visual Question Answering

# Construct the input payload based on the action's requirements
payload = {
    "image": "https://replicate.delivery/pbxt/M9rZEPYihLWYsHIHMw32oStoi20o9AShqpGzy7WRwz4rHQpp/rococo.jpg",
    "prompt": "Which era does this piece belong to? Give details about the era.",
    "temperature": 0.7,
    "maximumTokens": 512,
    "topProbability": 0.95
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload} # Hypothetical structure
    )
    response.raise_for_status()  # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this snippet, replace YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key. The action ID and input payload must be correctly structured to ensure a successful call.

Conclusion

The Perform Visual Question Answering action from the Llama 3.2-Vision 90B spec opens up exciting possibilities for integrating visual reasoning capabilities into your applications. By leveraging this Cognitive Action, you can enhance user experiences through intelligent image analysis and response generation. Consider exploring various use cases, from educational tools to art analysis applications, to fully utilize the potential of this powerful action.

← Back to All Articles