Scrape Multi-Page Websites Recursively with Google Sheets Storage

This n8n workflow automates the process of recursively scraping data from multi-page websites and storing the extracted information in a Google Sheet. It's designed to efficiently navigate through linked pages, extract specific data, and centralize it for analysis or further processing.

What it does

1. Manual Trigger: Initiates the workflow upon manual execution.
2. HTTP Request: Fetches the HTML content of the initial web page.
3. HTML (Scrape Initial Page): Extracts specific data points from the initial page's HTML.
4. If: Checks if there are more pages to scrape (e.g., a "next page" link or pagination indicator).
   • If True (More Pages):
     • HTTP Request (Next Page): Fetches the HTML content of the next page.
     • HTML (Scrape Next Page): Extracts data from the subsequent page.
     • Merge: Combines the data from the current page with data from previous pages.
     • Edit Fields (Set): Prepares the combined data for storage, potentially cleaning or reformatting it.
     • If: Loops back to check for more pages, creating a recursive scraping mechanism.
   • If False (No More Pages):
     • Aggregate: Collects all scraped data into a single structure.
     • Split Out: Processes the aggregated data, potentially splitting it into individual items.
     • Google Sheets: Appends the final, structured data to a specified Google Sheet.

Prerequisites/Requirements

• n8n Instance: A running instance of n8n.
• Google Account: A Google account with access to Google Sheets.
• Google Sheets Credential: An n8n credential configured for Google Sheets to allow the workflow to write data.

Setup/Usage

1. Import the Workflow: Import the provided JSON into your n8n instance.
2. Configure Credentials:
   • Set up a Google Sheets credential in n8n.
3. Configure HTTP Request Nodes:
   • Update the "HTTP Request" nodes with the starting URL of the website you want to scrape.
   • Adjust the "HTTP Request (Next Page)" node to correctly identify and fetch subsequent pages (e.g., by extracting the "next page" URL from the initial scrape).
4. Configure HTML Nodes:
   • Modify the "HTML (Scrape Initial Page)" and "HTML (Scrape Next Page)" nodes to define the CSS selectors for the data you wish to extract from the web pages (e.g., product names, prices, descriptions, links to other pages).
5. Configure If Nodes:
   • Adjust the conditions in the "If" nodes to accurately detect the presence of more pages to scrape. This typically involves checking for the existence of a "next page" link or a specific pagination element.
6. Configure Google Sheets Node:
   • Specify the Spreadsheet ID and Sheet Name where you want to store the scraped data.
   • Ensure the column headers in your Google Sheet match the keys of the data being output by the "Edit Fields (Set)" node.
7. Activate and Execute:
   • Save and activate the workflow.
   • Click "Execute Workflow" on the "Manual Trigger" node to run it.