Detect Open-Vocabulary Objects with the OWL-ViT Model: A Developer's Guide

23 Apr 2025
Detect Open-Vocabulary Objects with the OWL-ViT Model: A Developer's Guide

Integrating advanced image analysis capabilities into your applications can significantly enhance user experience and functionality. The adirik/owlvit-base-patch32 spec provides developers with a powerful Cognitive Action for detecting objects in images using the OWL-ViT model. This action enables zero-shot object detection, allowing you to query images with text descriptions and receive detailed results, including bounding box coordinates and confidence scores. Below, we'll explore how to leverage this action effectively in your applications.

Prerequisites

Before you can utilize the Cognitive Actions in the adirik/owlvit-base-patch32 spec, ensure you have the following:

  • An API key for the Cognitive Actions platform.
  • Familiarity with JSON structure and basic HTTP requests.
  • Understanding of how to authenticate against the Cognitive Actions API by passing your API key in the request headers.

Cognitive Actions Overview

Detect Open-Vocabulary Objects

Purpose: This action performs zero-shot object detection on images using the OWL-ViT model, which combines CLIP and vision transformer backbones for open-vocabulary recognition. You can upload an image and query it with text descriptions of objects to detect, adjusting the confidence thresholds as needed.

Category: Object Detection

Input

The input for this action requires a structured JSON object with the following fields:

  • image (string): A valid URI pointing to the input image to be processed.
  • query (string): A comma-separated list of object names to detect within the image (e.g., "human face, rocket, star-spangled banner, nasa badge").
  • threshold (number, optional): A confidence level threshold for object detection, ranging from 0 to 1. The default value is 0.1.
  • showVisualization (boolean, optional): A flag indicating whether to display bounding boxes around detected objects. The default is true.

Example Input:

{
  "image": "https://replicate.delivery/pbxt/JhlycB8ScNVrMu0ke1Xlg09ajbsmMfp4TK19JXpnYq6GrHK8/astronaut.png",
  "query": "human face, rocket, star-spangled banner, nasa badge",
  "threshold": 0.11,
  "showVisualization": true
}

Output

The action returns a JSON object containing:

  • json_data: A list of detected objects, each with a bounding box (bbox), a label, and a confidence score.
  • result_image: A URL to an image showing the original image with visualizations (bounding boxes) of detected objects.

Example Output:

{
  "json_data": {
    "objects": [
      {
        "bbox": [180, 71, 271, 178],
        "label": "human face",
        "confidence": 0.35713595151901245
      },
      {
        "bbox": [1, 1, 105, 509],
        "label": "star-spangled banner",
        "confidence": 0.13790424168109894
      },
      {
        "bbox": [350, -1, 468, 288],
        "label": "rocket",
        "confidence": 0.2110234647989273
      },
      {
        "bbox": [129, 348, 206, 427],
        "label": "nasa badge",
        "confidence": 0.28099769353866577
      },
      {
        "bbox": [277, 338, 327, 380],
        "label": "nasa badge",
        "confidence": 0.1195005401968956
      }
    ]
  },
  "result_image": "https://assets.cognitiveactions.com/invocations/9cbac741-ecdf-44ba-a893-f14770c1ff0d/29daab04-c23f-4381-82bb-83017a6194f5.png"
}

Conceptual Usage Example (Python)

Here’s how you might call the Detect Open-Vocabulary Objects action using Python:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute" # Hypothetical endpoint

action_id = "a65184c6-901f-4498-b47a-347d57cf9b89" # Action ID for Detect Open-Vocabulary Objects

# Construct the input payload based on the action's requirements
payload = {
    "image": "https://replicate.delivery/pbxt/JhlycB8ScNVrMu0ke1Xlg09ajbsmMfp4TK19JXpnYq6GrHK8/astronaut.png",
    "query": "human face, rocket, star-spangled banner, nasa badge",
    "threshold": 0.11,
    "showVisualization": true
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload} # Hypothetical structure
    )
    response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this code snippet, replace YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key. The payload structure follows the required input schema, and the action ID is specified for invoking the correct action.

Conclusion

The Detect Open-Vocabulary Objects action from the adirik/owlvit-base-patch32 spec provides a robust solution for object detection in images, enabling developers to enhance their applications with advanced image recognition capabilities. By leveraging this action, you can automate the detection of various objects with adjustable confidence thresholds, making it a powerful tool for diverse use cases. Consider experimenting with different queries and threshold values to explore the full potential of this action!

← Back to All Articles