Enhance Your Applications with Seamless Speech Interaction Using LLaMA-Omni Cognitive Actions

22 Apr 2025
Enhance Your Applications with Seamless Speech Interaction Using LLaMA-Omni Cognitive Actions

The LLaMA-Omni API provides developers with powerful Cognitive Actions designed to facilitate seamless speech interactions. Built on the advanced LLaMA model architecture, these actions enable high-quality text and speech responses with minimal latency. Integrating these pre-built actions into your applications can enhance user experiences by allowing for natural and intuitive communication.

Prerequisites

Before diving into the Cognitive Actions, ensure you have the following:

  • An API key for accessing the LLaMA-Omni Cognitive Actions platform.
  • Basic knowledge of making API calls and handling JSON data.
  • A suitable environment for running Python code.

Authentication typically involves passing your API key in the headers of your requests to the Cognitive Actions endpoint.

Cognitive Actions Overview

Enable Seamless Speech Interaction

The Enable Seamless Speech Interaction action is designed to facilitate high-quality speech-to-text conversions using the LLaMA-Omni model. This action processes audio inputs and generates both text and audio responses, ensuring low-latency interactions for applications that require real-time feedback.

Input

The action requires the following input fields:

  • inputAudio (required): A URI pointing to the input audio file to be processed.
  • prompt (optional): A string guiding the model on how to respond. The default prompt is "Please directly answer the questions in the user's speech."
  • temperature (optional): A value controlling the randomness of the response (0 to 1). Lower values yield more deterministic outputs.
  • maxNewTokens (optional): The maximum number of tokens to generate in the response (default is 256).
  • topProbability (optional): A value determining the diversity of the output when the temperature is greater than 0 (0 to 1).

Example Input:

{
  "prompt": "Please directly answer the questions in the user's speech",
  "inputAudio": "https://replicate.delivery/pbxt/LfbWz5nAdlqDatmo2feweGHjcVyJHdQhqZYRNHqfJ7EyKxXa/helpful_base_1.wav",
  "temperature": 0,
  "maxNewTokens": 256,
  "topProbability": 0
}

Output

The action typically returns a response containing:

  • text: The generated text response based on the input audio.
  • audio: A URI pointing to the audio file of the generated response.

Example Output:

{
  "text": "The origin of US state names is varied, but most were named by European explorers and settlers. Many were named after Native American tribes, Spanish and Mexican cities, or royal figures. Some states were also named after natural features, like rivers or mountains.",
  "audio": "https://assets.cognitiveactions.com/invocations/897fe694-ba11-462e-bb01-5fad6c6d15bd/37db20ba-80d3-49b1-90d5-23c3fd411597.wav"
}

Conceptual Usage Example (Python)

Here’s how you can conceptually use the Enable Seamless Speech Interaction action in Python:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute"  # Hypothetical endpoint

action_id = "0d298d27-a899-4415-a645-64252a69a713"  # Action ID for Enable Seamless Speech Interaction

# Construct the input payload based on the action's requirements
payload = {
    "prompt": "Please directly answer the questions in the user's speech",
    "inputAudio": "https://replicate.delivery/pbxt/LfbWz5nAdlqDatmo2feweGHjcVyJHdQhqZYRNHqfJ7EyKxXa/helpful_base_1.wav",
    "temperature": 0,
    "maxNewTokens": 256,
    "topProbability": 0
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload}  # Hypothetical structure
    )
    response.raise_for_status()  # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this example, replace the YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key. The action ID and input payload are structured based on the requirements of the Enable Seamless Speech Interaction action.

Conclusion

Integrating the LLaMA-Omni Cognitive Actions into your applications provides a straightforward approach to enhancing user interactions through speech. With the ability to process audio inputs and generate immediate, relevant responses, you can create engaging and responsive applications. Next, consider exploring additional use cases like voice-activated assistants or customer service bots to leverage these capabilities fully!

← Back to All Articles