machine-learning

Generate Multi-Modal Predictions with Fuyu 8b

25 Apr 2025
Generate Multi-Modal Predictions with Fuyu 8b

In today's fast-paced digital landscape, the ability to generate intelligent insights from diverse data types is becoming increasingly crucial. The Fuyu 8b model offers developers a powerful solution for generating multi-modal predictions by seamlessly integrating text and image inputs. This advanced transformer model, developed by Adept AI, excels at processing complex data to deliver meaningful outputs. By leveraging the capabilities of Fuyu 8b, developers can automate tasks, enhance user experiences, and gain deeper insights from their data.

Common use cases for Fuyu 8b's multi-modal predictions include applications in educational tools, where users might query data visualizations, or in customer service settings, where images of products can be analyzed alongside textual inquiries. This dual capability simplifies workflows and enhances the accuracy of generated information, making it an invaluable asset for developers.

Generate Multi-Modal Predictions

The "Generate Multi-Modal Predictions" action enables developers to harness the full potential of the Fuyu 8b model by generating predictions that combine both text and image data. This action solves the problem of needing to interpret and analyze information from multiple sources simultaneously, allowing for richer and more contextual insights.

Input Requirements

To utilize this action, you need to provide a composite request that includes:

  • Image: A valid URI pointing to an image resource. This image serves as the visual input for the model.
  • Prompt: A textual query that defines the context of the request. This helps the model understand what information you seek from the image.
  • Max New Tokens (optional): An integer specifying the maximum number of new tokens that can be generated in the response, with a range from 0 to 2048 (default is 512).

Example Input:

{
  "image": "https://replicate.delivery/pbxt/JjK2zhdhMpdevuSR7POm4X64qa2fVWv8miI4NBlkoHWVPmpD/chart.png",
  "prompt": "What is the highest life expectancy at birth of male?",
  "maxNewTokens": 512
}

Expected Output

The output will be a text response generated by the model, providing the answer to the given prompt based on the analysis of the image.

Example Output:

The life expectancy at birth of males in 2018 is 80.7.

Use Cases for this Action

This action is particularly useful in scenarios such as:

  • Data Analysis: Analyzing visual data representations like charts or graphs and generating insights based on user queries.
  • Interactive Learning: Creating educational tools that respond to student inquiries with both visual and textual data, enhancing comprehension.
  • Customer Support: Enabling support systems to answer customer questions about products by interpreting images and providing relevant information.
import requests
import json

# Replace with your actual Cognitive Actions API key and endpoint
# Ensure your environment securely handles the API key
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
# This endpoint URL is hypothetical and should be documented for users
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute"

action_id = "a3542640-cb7f-4fd5-8cc6-3b240be9f020" # Action ID for: Generate Multi-Modal Predictions

# Construct the exact input payload based on the action's requirements
# This example uses the predefined example_input for this action:
payload = {
  "image": "https://replicate.delivery/pbxt/JjK2zhdhMpdevuSR7POm4X64qa2fVWv8miI4NBlkoHWVPmpD/chart.png",
  "prompt": "What is the highest life expectancy at birth of male?",
  "maxNewTokens": 512
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json",
    # Add any other required headers for the Cognitive Actions API
}

# Prepare the request body for the hypothetical execution endpoint
request_body = {
    "action_id": action_id,
    "inputs": payload
}

print(f"--- Calling Cognitive Action: {action.name or action_id} ---")
print(f"Endpoint: {COGNITIVE_ACTIONS_EXECUTE_URL}")
print(f"Action ID: {action_id}")
print("Payload being sent:")
print(json.dumps(request_body, indent=2))
print("------------------------------------------------")

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json=request_body
    )
    response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully. Result:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body (non-JSON): {e.response.text}")
    print("------------------------------------------------")

In conclusion, the Fuyu 8b model's ability to generate multi-modal predictions opens up a world of possibilities for developers. By integrating text and image data, you can create applications that are not only smarter but also more intuitive and responsive to user needs. Whether you're enhancing educational tools, improving customer service, or conducting in-depth data analysis, Fuyu 8b can significantly streamline your processes and elevate the user experience. To get started, ensure you have your Cognitive Actions API key and explore the potential of multi-modal predictions in your next project!

← Back to All Articles