Vision-Based AI Agent Scraper with Google Sheets, ScrapingBee, and Gemini

This n8n workflow orchestrates a powerful AI agent to perform vision-based web scraping, leveraging Google Sheets for input, ScrapingBee for web content retrieval, and Google Gemini for intelligent analysis. It's designed to automate the process of extracting and processing information from web pages based on specific prompts.

What it does

This workflow automates the following steps:

1. Triggers Manually or via Another Workflow: The workflow can be initiated manually with a single click or programmatically by another n8n workflow.
2. Reads Input from Google Sheets: It fetches a list of URLs and associated prompts from a specified Google Sheet. Each row in the sheet should contain a URL to scrape and a prompt for the AI agent to follow.
3. Prepares Data for AI Agent: It transforms the data from Google Sheets into a format suitable for the AI agent, specifically creating a url and prompt field for each item.
4. Initializes AI Agent: It sets up an AI Agent using the Google Gemini Chat Model. This agent is equipped with a "Call n8n Workflow Tool" to interact with the ScrapingBee API for web scraping.
5. Performs Vision-Based Scraping: For each URL and prompt, the AI Agent uses its "Call n8n Workflow Tool" to:
   • Make an HTTP request to the ScrapingBee API, providing the target URL and instructing it to render the page with JavaScript and return the full HTML.
   • Process the retrieved HTML content using the Google Gemini Chat Model based on the provided prompt. The prompt guides the AI on what information to extract or analyze from the page's content.
6. Parses Structured Output: The AI Agent's response is then passed through a Structured Output Parser to extract the relevant information in a predefined JSON format.
7. Formats Output as Markdown: The extracted information is formatted into a human-readable Markdown string.
8. Splits Output: The workflow splits the output, likely to process each scraped item individually or prepare it for subsequent actions (though no further nodes are connected in the provided JSON).

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running instance of n8n.
• Google Sheets Account: Configured credentials for Google Sheets to read input data.
• ScrapingBee Account: An API key for ScrapingBee to perform web scraping.
• Google Gemini API Key: Credentials for the Google Gemini Chat Model.
• n8n Credentials: Appropriate n8n API credentials for the "Call n8n Workflow Tool" to execute sub-workflows (if the ScrapingBee HTTP Request is encapsulated in a separate workflow).

Setup/Usage

1. Import the Workflow: Import the provided JSON into your n8n instance.
2. Configure Credentials:
   • Update the "Google Sheets" node with your Google Sheets credentials and specify the spreadsheet and sheet name containing your URLs and prompts.
   • Configure the "Google Gemini Chat Model" node with your Google Gemini API key.
   • Set up the "HTTP Request" node (within the "Call n8n Workflow Tool" if it's a sub-workflow) with your ScrapingBee API key.
   • Ensure the "Call n8n Workflow Tool" has access to the necessary n8n API credentials to execute the scraping logic.
3. Define Input in Google Sheets: Create a Google Sheet with at least two columns: one for the url to be scraped and another for the prompt that guides the AI agent on what to look for on that URL.
4. Activate the Workflow: Enable the workflow in n8n.
5. Execute the Workflow:
   • Click "Execute workflow" in the n8n editor to run it manually.
   • Alternatively, trigger it from another workflow using the "Execute Workflow Trigger" node.

The workflow will then process each row in your Google Sheet, scrape the specified URL, and use the AI agent to extract information based on your prompt, outputting the results in Markdown format.