Domain-Specific Web Content Crawler with Depth Control and Text Extraction

This n8n workflow provides a powerful and flexible solution for crawling web content, extracting text, and controlling the crawling depth. It's ideal for gathering specific information from websites, building datasets for analysis, or monitoring changes on particular pages.

What it does

This workflow automates the following steps:

1. Receives a Webhook Trigger: The workflow starts when it receives an HTTP POST request to its webhook URL. This request should include the url to start crawling from and an optional depth parameter to control how many levels deep the crawler should go.
2. Initializes Crawling Parameters: Sets default values for depth (if not provided) and crawledUrls (to keep track of visited URLs).
3. Loops Through URLs: It enters a loop that processes URLs in batches.
4. Checks Crawling Depth: For each URL, it verifies if the current crawling depth is within the specified limit.
5. Fetches Web Page Content: If the depth limit is not exceeded, it makes an HTTP GET request to fetch the HTML content of the URL.
6. Extracts Text and Links: It then uses an HTML node to extract all visible text content and all href attributes (links) from the fetched page.
7. Filters and Prepares New URLs: It filters out invalid or already crawled links and prepares the new URLs for the next iteration of the loop, incrementing the depth for each.
8. Merges Results: After processing a batch, it merges the extracted text and new URLs back into the main flow.
9. Responds to Webhook: Once all crawling is complete, it responds to the initial webhook with the collected text content.

Prerequisites/Requirements

• n8n Instance: A running n8n instance to import and execute the workflow.
• Basic Understanding of Webhooks: Knowledge of how to send HTTP POST requests to trigger the workflow.

Setup/Usage

1. Import the Workflow:
   • Copy the provided JSON code.
   • In your n8n instance, click "New" to create a new workflow.
   • Go to "File" > "Import from JSON" and paste the JSON code.
   • Click "Import".
2. Activate the Workflow:
   • Ensure the workflow is activated by toggling the "Active" switch in the top right corner of the n8n editor.
3. Get the Webhook URL:
   • The "Webhook" node (Node ID: 47) will display a unique URL. Copy this URL.
4. Trigger the Workflow:
   • Send an HTTP POST request to the copied Webhook URL.
   • The request body should be JSON and contain at least a url field.
   • Optionally, include a depth field to control the crawling depth (e.g., {"url": "https://example.com", "depth": 2}). If depth is omitted, it defaults to 1.

Example cURL command to trigger the workflow:

curl -X POST \
  YOUR_WEBHOOK_URL \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://n8n.io/blog/",
    "depth": 2
  }'

The workflow will return the extracted text content from the crawled pages in its response.