image-processing

Enhance Image Understanding with Panoptic Scene Graphs

26 Apr 2025
Enhance Image Understanding with Panoptic Scene Graphs

Openpsg offers a powerful set of Cognitive Actions designed for image processing, with a focus on generating detailed scene graphs from visual inputs. The "Generate Panoptic Scene Graph" action stands out as a remarkable tool for developers looking to enhance image understanding in their applications. By leveraging panoptic segmentation, this action addresses common challenges in scene graph generation, such as coarse localization and duplicate groundings, ensuring a high degree of accuracy and clarity in predicate definitions.

This API can significantly speed up the development process for applications that require a deep understanding of image content. Use cases range from improving computer vision tasks in robotics to enriching multimedia content analysis. Whether you're building an app that recognizes objects in images, creating an interactive gaming experience, or developing advanced surveillance systems, the ability to generate precise scene graphs is invaluable.

Prerequisites

To get started, you'll need an Openpsg Cognitive Actions API key and a basic understanding of making API calls.

Generate Panoptic Scene Graph

The "Generate Panoptic Scene Graph" action creates a comprehensive scene graph from an image input, effectively addressing issues like inaccurate localization and redundant groundings. This action is categorized under image processing and is essential for applications that require detailed contextual information from images.

Input Requirements

This action requires the following input:

  • image: A URI string pointing to the input image, which is mandatory for processing.
  • numberOfRelations: An integer that specifies how many relations will be generated in the scene graph, ranging from 1 to 20, with a default value of 5.

Example Input:

{
  "image": "https://replicate.delivery/mgxm/f10e3e9e-ef22-4957-bf25-d2ca40f53f8a/friends.jpeg",
  "numberOfRelations": 5
}

Expected Output

The output will consist of a series of images that visually represent the scene graph generated from the input. Each output image will illustrate the relationships between the various objects identified in the original image.

Example Output:

[
  {
    "image": "https://assets.cognitiveactions.com/invocations/13f598ae-ceb9-4695-a444-391f55c521f5/e600d25c-f43a-4d5d-b692-04e77bb21afe.png"
  },
  {
    "image": "https://assets.cognitiveactions.com/invocations/13f598ae-ceb9-4695-a444-391f55c521f5/1b516007-8bb8-42e3-87dd-7a45d65bf647.png"
  },
  {
    "image": "https://assets.cognitiveactions.com/invocations/13f598ae-ceb9-4695-a444-391f55c521f5/cacc5443-6e8e-474e-a1e2-72f82c34cb6d.png"
  },
  {
    "image": "https://assets.cognitiveactions.com/invocations/13f598ae-ceb9-4695-a444-391f55c521f5/81bc758a-f219-4232-a5c6-0cdace89a4e6.png"
  },
  {
    "image": "https://assets.cognitiveactions.com/invocations/13f598ae-ceb9-4695-a444-391f55c521f5/aff87178-a5b9-4505-9911-890925a18fef.png"
  }
]

Use Cases for this Action:

  • Robotics and Automation: Enhance the perception capabilities of robots by providing them with detailed scene graphs, enabling better navigation and interaction with their environment.
  • Content Analysis: In applications like photo management or social media, use scene graphs to automatically tag and categorize images based on their content.
  • Gaming: Create more immersive gaming experiences by using scene graphs to define relationships between objects, thereby enriching gameplay dynamics.
  • Surveillance Systems: Improve the accuracy of object detection and tracking in surveillance footage, facilitating better incident analysis and response.
import requests
import json

# Replace with your actual Cognitive Actions API key and endpoint
# Ensure your environment securely handles the API key
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
# This endpoint URL is hypothetical and should be documented for users
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute"

action_id = "1edefe7a-12eb-4157-bc02-bf97aa7de640" # Action ID for: Generate Panoptic Scene Graph

# Construct the exact input payload based on the action's requirements
# This example uses the predefined example_input for this action:
payload = {
  "image": "https://replicate.delivery/mgxm/f10e3e9e-ef22-4957-bf25-d2ca40f53f8a/friends.jpeg",
  "numberOfRelations": 5
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json",
    # Add any other required headers for the Cognitive Actions API
}

# Prepare the request body for the hypothetical execution endpoint
request_body = {
    "action_id": action_id,
    "inputs": payload
}

print(f"--- Calling Cognitive Action: {action.name or action_id} ---")
print(f"Endpoint: {COGNITIVE_ACTIONS_EXECUTE_URL}")
print(f"Action ID: {action_id}")
print("Payload being sent:")
print(json.dumps(request_body, indent=2))
print("------------------------------------------------")

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json=request_body
    )
    response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully. Result:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body (non-JSON): {e.response.text}")
    print("------------------------------------------------")

Conclusion

The "Generate Panoptic Scene Graph" action from Openpsg empowers developers to extract deeper insights from images by generating detailed scene graphs. This capability not only enhances the understanding of visual content but also opens up a multitude of applications across various domains. By integrating this action into your projects, you can streamline processes, improve accuracy, and ultimately deliver richer experiences to your users. Explore the possibilities with Openpsg and take your image processing capabilities to the next level!

← Back to All Articles