Unlock the Power of Web Scraping with myaiteam2/website-scrapper

24 Apr 2025
Unlock the Power of Web Scraping with myaiteam2/website-scrapper

Web scraping is a vital technique for developers looking to extract data from websites for various applications, from data analysis to content aggregation. The myaiteam2/website-scrapper API provides a powerful Cognitive Action designed to streamline this process using BeautifulSoup, a popular Python library for web scraping. By leveraging this API, developers can easily access data from websites while being mindful of potential scraping restrictions some sites may impose.

Prerequisites

To get started with the Cognitive Actions from the myaiteam2/website-scrapper, you will need:

  • An API key from the Cognitive Actions platform.
  • Basic knowledge of Python and HTTP requests.

Authentication typically involves passing your API key in the headers of your API requests.

Cognitive Actions Overview

Perform Web Scraping with BeautifulSoup

This action allows developers to scrape data from specified URLs using the BeautifulSoup library. It's important to note that some websites may have measures in place to block scraping attempts.

Input

The input schema for this action requires a single field:

  • url (string, required): The URL of the website to scrape. Make sure the URL is valid and accessible.

Example Input:

{
  "url": "https://myaiteam.com"
}

Output

The output of this action typically includes:

  • emails (array): A list of extracted email addresses found on the page.
  • image_list (array): A list of images extracted from the page, including their src and alt attributes.
  • html_content (string): The raw HTML content of the scraped page.
  • text_content (string): The plain text content extracted from the HTML.

Example Output:

{
  "emails": [
    "flags@2x.png"
  ],
  "image_list": [
    {
      "alt": "",
      "src": "https://images.leadconnectorhq.com/image/f_webp/q_80/r_1200/u_https://assets.cdn.filesafe.space/ROFxNV6LImNajmA6DHCV/media/666a36bf6315f72370e91b7c.png",
      "label": "image"
    }
  ],
  "html_content": "<!DOCTYPE html><html lang=\"en\"><head>...</head><body>...</body></html>",
  "text_content": "MyAiTeam - Ai Coding Software For Website Builders..."
}

Conceptual Usage Example (Python)

Here's a conceptual example of how you might call the Perform Web Scraping with BeautifulSoup action using Python:

import requests
import json

# Replace with your Cognitive Actions API key and endpoint
COGNITIVE_ACTIONS_API_KEY = "YOUR_COGNITIVE_ACTIONS_API_KEY"
COGNITIVE_ACTIONS_EXECUTE_URL = "https://api.cognitiveactions.com/actions/execute"  # Hypothetical endpoint

action_id = "025f7276-420f-445f-a550-f2ad40407828"  # Action ID for Perform Web Scraping with BeautifulSoup

# Construct the input payload based on the action's requirements
payload = {
    "url": "https://myaiteam.com"
}

headers = {
    "Authorization": f"Bearer {COGNITIVE_ACTIONS_API_KEY}",
    "Content-Type": "application/json"
}

try:
    response = requests.post(
        COGNITIVE_ACTIONS_EXECUTE_URL,
        headers=headers,
        json={"action_id": action_id, "inputs": payload}  # Hypothetical structure
    )
    response.raise_for_status()  # Raise an exception for bad status codes (4xx or 5xx)

    result = response.json()
    print("Action executed successfully:")
    print(json.dumps(result, indent=2))

except requests.exceptions.RequestException as e:
    print(f"Error executing action {action_id}: {e}")
    if e.response is not None:
        print(f"Response status: {e.response.status_code}")
        try:
            print(f"Response body: {e.response.json()}")
        except json.JSONDecodeError:
            print(f"Response body: {e.response.text}")

In this example, replace YOUR_COGNITIVE_ACTIONS_API_KEY with your actual API key. The payload variable constructs the input JSON needed for the request, and the code handles the response, printing the results or any errors encountered.

Conclusion

Integrating the myaiteam2/website-scrapper Cognitive Actions into your applications can significantly simplify the web scraping process, allowing you to focus on analyzing and utilizing the gathered data effectively. Consider exploring additional use cases, such as automating data collection for research purposes or monitoring website changes. With the power of BeautifulSoup and the convenience of this API, your web scraping tasks can be accomplished with ease.

← Back to All Articles