Ground Multimodal Language Models with lucataco/kosmos-2 Cognitive Actions

23 Apr 2025
Ground Multimodal Language Models with lucataco/kosmos-2 Cognitive Actions

In today's rapidly evolving landscape of artificial intelligence, the ability to integrate multimodal capabilities into applications is essential. The lucataco/kosmos-2 spec offers developers powerful Cognitive Actions that allow the grounding of large language models to real-world contexts using images. This post will guide you through the "Ground Multimodal Language Models" action, detailing its functionality, input requirements, output structure, and how you can leverage it in your applications.

Prerequisites

To get started with Cognitive Actions, ensure you have the following:

  • An API key for the Cognitive Actions platform, which will be used for authentication.
  • Basic knowledge of making API requests and handling JSON data.

Authentication typically involves passing your API key in the request headers, allowing you to securely interact with the Cognitive Actions endpoint.

Cognitive Actions Overview

Ground Multimodal Language Models

This action utilizes Kosmos-2 to ground multimodal large language models to the world. It accepts an image URI and optionally outputs images with bounding boxes, providing a description that can be tailored for either brevity or detail.

Input

The input for this action is structured as follows:

{
  "image": "https://replicate.delivery/pbxt/JoIS31y9Oy2m04rBBICdzXUw7WleL1uCP6dyV5TeKTft2jjB/snowman.png",
  "visualOutput": true,
  "descriptionType": "Detailed"
}
  • image (required): A string representing the URI of the input image. This field is mandatory.
  • visualOutput (optional): A boolean value indicating whether the output should include the image with bounding boxes. The default value is true.
  • descriptionType (optional): A string that specifies the level of detail in the description, with options for "Brief" or "Detailed". The default is "Brief".

Output

The output from this action typically includes:

{
  "img": "https://assets.cognitiveactions.com/invocations/0afffc95-f410-4af2-8b77-57392955895e/4daa264e-f38f-48f4-85af-5fdaf0980f99.jpg",
  "text": "Describe this image in detail: The image features a snowman sitting by a campfire in the snow..."
}
  • img: A string providing the URI of the output image with bounding boxes.
  • text: A detailed description of the input image, summarizing key elements and their relationships.

Conceptual Usage Example (Python)

Here’s how you might invoke this action using Python:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute"  # Hypothetical endpoint

action_id = "d057a686-f4a6-4a7b-96c9-d2e8ef414257"  # Action ID for Ground Multimodal Language Models

# Construct the input payload based on the action's requirements
payload = {
    "image": "https://replicate.delivery/pbxt/JoIS31y9Oy2m04rBBICdzXUw7WleL1uCP6dyV5TeKTft2jjB/snowman.png",
    "visualOutput": True,
    "descriptionType": "Detailed"
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload}  # Hypothetical structure
    )
    response.raise_for_status()  # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this code snippet, replace the placeholder for your API key and ensure the action ID corresponds to the desired action. The input payload is structured according to the specifications, and you can easily modify it based on your needs.

Conclusion

The Ground Multimodal Language Models action from the lucataco/kosmos-2 spec provides a robust means of integrating advanced image analysis and language understanding into your applications. By leveraging this action, developers can create applications that interpret images and convey detailed descriptions, opening up new possibilities for user interaction and content understanding. Explore the potential of cognitive actions in your projects and enhance your applications with cutting-edge AI capabilities!

← Back to All Articles