Elevate Your Evaluation Process with tomasmcm/prometheus-13b-v1.0 Cognitive Actions

25 Apr 2025
Elevate Your Evaluation Process with tomasmcm/prometheus-13b-v1.0 Cognitive Actions

In today's digital landscape, providing detailed feedback on content can significantly enhance user experience and understanding. The tomasmcm/prometheus-13b-v1.0 API offers a powerful Cognitive Action that allows developers to evaluate long-form responses with precision. By leveraging the capabilities of the Prometheus-13B model, you can provide feedback based on a custom rubric, ensuring that your evaluations are not only accurate but also sensitive to the nuances of context, creativity, and emotional intelligence.

Prerequisites

To get started with the Cognitive Actions from the tomasmcm/prometheus-13b-v1.0 API, you will need to have an API key for authentication. This key should be passed in the headers of your requests to access the service securely. Ensure that you have the necessary setup in place before integrating the actions into your applications.

Cognitive Actions Overview

Evaluate Response with Prometheus

The Evaluate Response with Prometheus action allows you to assess the quality of textual responses based on a detailed scoring rubric. This action is particularly useful for applications that require nuanced evaluations, such as grading responses or providing feedback in educational tools.

  • Category: Text Classification
  • Purpose: Evaluates long-form responses, offering feedback and scoring based on criteria like readability, cultural sensitivity, and creativity.

Input

The action requires a JSON object as input, with the following schema:

{
  "prompt": "string (required)",
  "maxTokens": "integer (optional, default 128)",
  "topK": "integer (optional, default -1)",
  "topP": "number (optional, default 0.95)",
  "temperature": "number (optional, default 0.8)",
  "presencePenalty": "number (optional, default 0)",
  "frequencyPenalty": "number (optional, default 0)",
  "stop": "string (optional)"
}

Example Input:

{
  "topK": -1,
  "topP": 0.95,
  "prompt": "###Task Description:\nAn instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.\n1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.\n3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)\"\n4. Please do not generate any other opening, closing, and explanations.\n\n###The instruction to evaluate:\nYou are a well-known psychiatrist who has a reputation for being empathetic and understanding. A client comes to you saying they've had a really hard day at work. They describe their boss as being overly critical and not listening to their ideas, which has left them feeling frustrated and undervalued. They also mention feeling overwhelmed with the workload and a growing sense of loneliness as they are new to the city. How do you respond to them to make them feel heard and understood, as well as offering helpful advice?\n\n###Response to evaluate:\nIt's indeed challenging to deal with a difficult boss and to carry a large workload, especially when you are new to a city and haven't established a supportive network. I would suggest taking a step back to think about your situation and perhaps speak to your boss regarding the issues you're facing. On the other hand, dealing with a large workload can be managed by prioritizing tasks, and maybe you can discuss it with your superiors too. In regards to feeling lonely, you might want to explore activities or groups that align with your interests to meet new people. And also, remember to take time for yourself to relax and destress. I'm here to assist you as we navigate through these issues.\n\n###Reference Answer (Score 5):\nI hear that you're dealing with quite a few challenging situations at once. It must be tough to have your ideas dismissed by your boss and to feel undervalued in your work environment. It's understandable that such experiences can lead to frustration and a feeling of being overwhelmed. It's important to acknowledge these feelings and not to dismiss them. Feeling heard and validated in our experiences can often be the first step towards resolving the challenges we face. So, I want you to know that your feelings are completely valid and it's okay to be frustrated and overwhelmed...",
  "maxTokens": 512,
  "temperature": 0.8,
  "presencePenalty": 0,
  "frequencyPenalty": 0
}

Output

The action typically returns a feedback string that evaluates the response along with a score. Here's an example of what you might receive:

Example Output:

The response accurately identifies the user's emotional context and offers solutions to the challenges the user faces... So the overall score is 3. [RESULT] 3

Conceptual Usage Example (Python)

Here's how you might invoke the Evaluate Response with Prometheus action using Python:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute" # Hypothetical endpoint

action_id = "a4a7c21f-ac55-474f-9281-76e7ca36325c" # Action ID for Evaluate Response with Prometheus

# Construct the input payload based on the action's requirements
payload = {
  "topK": -1,
  "topP": 0.95,
  "prompt": "###Task Description:\n... (full prompt as per example input)",
  "maxTokens": 512,
  "temperature": 0.8,
  "presencePenalty": 0,
  "frequencyPenalty": 0
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload} # Hypothetical structure
    )
    response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this example, the action_id is set for the Evaluate Response with Prometheus action. The payload is structured according to the required input schema, and the API key is included in the request header for authentication.

Conclusion

The tomasmcm/prometheus-13b-v1.0 Cognitive Action for evaluating responses offers an effective way to provide nuanced and detailed feedback in various applications. By integrating this action, developers can enhance the quality of interactions in educational tools, customer support, and more. Start exploring the capabilities of this powerful model today, and elevate the way you evaluate and respond to user-generated content!

← Back to All Articles