n8n Website Scraper for LLMs with Airtop

This n8n workflow provides a robust solution for scraping website content, processing it, and storing it in Google Sheets, with an optional integration for Airtop. It's designed to prepare web content for use with Large Language Models (LLMs) or for general data collection and analysis.

What it does

This workflow automates the following steps:

1. Triggers on Form Submission: The workflow is initiated when a form is submitted, likely containing the URL of the website to be scraped.
2. Processes Input Data: An "Edit Fields" (Set) node prepares the incoming data, potentially extracting the target URL and other parameters.
3. Executes Custom Code: A "Code" node runs custom JavaScript, which is likely responsible for the actual web scraping logic. This node would fetch the content from the provided URL.
4. Conditional Logic for Airtop: An "If" node checks a condition, possibly to determine if the scraped data should be sent to Airtop.
5. Integrates with Airtop (Conditional): If the condition is met, the scraped data is sent to Airtop, likely for further processing or storage within the Airtop platform.
6. Stores Data in Google Sheets: Regardless of the Airtop integration, the scraped and processed data is then written to a specified Google Sheet, providing a centralized repository for the web content.
7. Generates Google Docs (Optional/Future): A "Google Docs" node is present but not connected in the provided JSON, suggesting a potential future enhancement for generating documents from the scraped content.
8. Sticky Note for Documentation: A "Sticky Note" is included for internal documentation within the workflow.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running n8n instance to import and execute the workflow.
• Google Sheets Account: Configured n8n credentials for Google Sheets to store the scraped data.
• Airtop Account (Optional): Configured n8n credentials for Airtop if you intend to use the Airtop integration path.
• Custom JavaScript for Scraping: The "Code" node requires custom JavaScript logic to perform the actual web scraping. This script will need to be written or provided based on the specific scraping requirements.
• n8n Form Trigger: The workflow is triggered by an n8n Form, which needs to be set up to collect the necessary input (e.g., website URL).

Setup/Usage

1. Import the Workflow: Download the workflow JSON and import it into your n8n instance.
2. Configure Credentials:
   • Set up your Google Sheets credentials in n8n.
   • If using Airtop, set up your Airtop credentials.
3. Configure the Form Trigger:
   • Access the "On form submission" node and configure its fields to accept the website URL and any other relevant parameters for scraping.
4. Customize the Code Node:
   • Open the "Code" node. You will need to write or paste your custom JavaScript code here to perform the web scraping. This code should take the URL from the previous node's output and return the desired scraped content.
5. Configure Google Sheets:
   • In the "Google Sheets" node, specify the Spreadsheet ID and Sheet Name where the scraped data should be stored. Map the incoming data fields to the appropriate columns in your Google Sheet.
6. Configure Airtop (Optional):
   • If you plan to use Airtop, configure the "Airtop" node with the necessary operation and resource, mapping the scraped data fields as required.
7. Activate the Workflow: Once configured, activate the workflow. You can then trigger it by submitting data to the n8n form.