HumationHuman + Automation
Back to Catalog

Ai website scraper & company intelligence

DIGITAL BIZ TECHDIGITAL BIZ TECH
923 views
2/3/2026
Google SheetsAutomationGoogle Cloud FirestoreData SyncDatabase
Official Page

AI Website Scraper & Company Intelligence

Description

This workflow automates the process of transforming any website URL into a structured, intelligent company profile.
It's triggered by a form, allowing a user to submit a website and choose between a "basic" or "deep" scrape.

The workflow extracts key information (mission, services, contacts, SEO keywords), stores it in a structured Supabase database, and archives a full JSON backup to Google Drive.
It also features a secondary AI agent that automatically finds and saves competitors for each company, building a rich, interconnected database of company intelligence.

Quick Implementation Steps

  1. Import the Workflow: Import the provided JSON file into your n8n instance.

  2. Install Custom Community Node:
    You must install the community node from:
    https://www.npmjs.com/package/n8n-nodes-crawl-and-scrape FIRECRAWL N8N Documentation https://docs.firecrawl.dev/developer-guides/workflow-automation/n8n

  3. Install Additional Nodes:
    n8n-nodes-crawl-and-scrape and n8n-nodes-mcp fire crawl mcp .

  4. Set up Credentials:
    Create credentials in n8n for FIRE CRAWL API,Supabase, Mistral AI, and Google Drive.

  5. Configure API Key (CRITICAL):

    • Open the Web Search tool node.
    • Go to Parameters → Headers and replace the hardcoded Tavily AI API key with your own.

  6. Configure Supabase Nodes:

    • Assign your Supabase credential to all Supabase nodes.
    • Ensure table names (e.g., companies, competitors) match your schema.

  7. Configure Google Drive Nodes:

    • Assign your Google Drive credential to the Google Drive2 and save to Google Drive1 nodes.
    • Select the correct Folder ID.

  8. Activate Workflow:
    Turn on the workflow and open the Webhook URL in the “On form submission” node to access the form.

What It Does

Form Trigger

Captures user input: “Website URL” and “Scraping Type” (basic or deep).

Scraping Router

A Switch node routes the flow:

  • Deep Scraping → AI-based MCP Firecrawler agent.
  • Basic Scraping → Crawlee node.

Deep Scraping (Firecrawl AI Agent)

  • Uses Firecrawl and Tavily Web Search.
  • Extracts a detailed JSON profile: mission, services, contacts, SEO keywords, etc.

Basic Scraping (Crawlee)

  • Uses Crawl and Scrape node to collect raw text.
  • A Mistral-based AI extractor structures the data into JSON.

Data Storage

  • Stores structured data in Supabase tables (companies, company_basicprofiles).
  • Archives a full JSON backup to Google Drive.

Automated Competitor Analysis

  • Runs after a deep scrape.
  • Uses Tavily web search to find competitors (e.g., from Crunchbase).
  • Saves competitor data to Supabase, linked by company_id.

Who's It For

  • Sales & Marketing Teams: Enrich leads with deep company info.
  • Market Researchers: Build structured, searchable company databases.
  • B2B Data Providers: Automate company intelligence collection.
  • Developers: Use as a base for RAG or enrichment pipelines.

Requirements

  • n8n instance (self-hosted or cloud)
  • Supabase Account: With tables like companies, competitors, social_links, etc.
  • Mistral AI API Key
  • Google Drive Credentials
  • Tavily AI API Key
  • (Optional) Custom Nodes:
    • n8n-nodes-crawl-and-scrape

How It Works

Flow Summary

  1. Form Trigger: Captures “Website URL” and “Scraping Type”.
  2. Switch Node:
    • deep → MCP Firecrawler (AI Agent).
    • basic → Crawl and Scrape node.
  3. Scraping & Extraction:
    • Deep path: Firecrawler → JSON structure.
    • Basic path: Crawlee → Mistral extractor → JSON.
  4. Storage:
    • Save JSON to Supabase.
    • Archive in Google Drive.
  5. Competitor Analysis (Deep Only):
    • Finds competitors via Tavily.
    • Saves to Supabase competitors table.
  6. End: Finishes with a No Operation node.

How To Set Up

  1. Import workflow JSON.
  2. Install community nodes (especially n8n-nodes-crawl-and-scrape from npm).
  3. Configure credentials (Supabase, Mistral AI, Google Drive).
  4. Add your Tavily API key.
  5. Connect Supabase and Drive nodes properly.
  6. Fix disconnected “basic” path if needed.
  7. Activate workflow.
  8. Test via the webhook form URL.

How To Customize

  • Change LLMs: Swap Mistral for OpenAI or Claude.
  • Edit Scraper Prompts: Modify system prompts in AI agent nodes.
  • Change Extraction Schema: Update JSON Schema in extractor nodes.
  • Fix Relational Tables: Add Items node before Supabase inserts for arrays (social links, keywords).
  • Enhance Automation: Add email/slack notifications, or replace form trigger with a Google Sheets trigger.

Add-ons

  • Automated Trigger: Run on new sheet rows.
  • Notifications: Email or Slack alerts after completion.
  • RAG Integration: Use the Supabase database as a chatbot knowledge source.

Use Case Examples

  • Sales Lead Enrichment: Instantly get company + competitor data from a URL.
  • Market Research: Collect and compare companies in a niche.
  • B2B Database Creation: Build a proprietary company dataset.

WORKFLOW IMAGE

Screenshot_22102025_152855_localhost.jpeg

Troubleshooting Guide

| Issue | Possible Cause | Solution | |-------|----------------|-----------| | Form Trigger 404 | Workflow not active | Activate the workflow | | Web Search Tool fails | Missing Tavily API key | Replace the placeholder key | | FIRECRAWLER / find competitor fails | Missing MCP node | Install n8n-nodes-mcp | | Basic scrape does nothing | Switch node path disconnected | Reconnect “basic” output | | Supabase node error | Wrong table/column names | Match schema exactly |

Need Help or More Workflows?

Want to customize this workflow for your business or integrate it with your existing tools?
Our team at Digital Biz Tech can tailor it precisely to your use case from automation logic to AI-powered enhancements.

Contact: shilpa.raju@digitalbiz.tech
For more such offerings, visit us: https://www.digitalbiz.tech

AI Website Scraper - Company Intelligence

This n8n workflow leverages AI to extract structured company intelligence from website content. It acts as a robust information extraction pipeline, capable of processing various inputs and storing the extracted data.

What it does

This workflow is designed to process incoming data, potentially from a form submission, and then use AI to extract specific information. Here's a breakdown of the steps:

  1. Receives Input: The workflow is triggered by an n8n Form Trigger, suggesting it's designed to receive data submitted through a web form.
  2. AI-Powered Information Extraction: An Information Extractor node (from Langchain) is used to intelligently parse the input data. This node is likely configured to identify and extract specific entities or data points related to company intelligence from unstructured text.
  3. Language Model: The Mistral Cloud Chat Model provides the underlying AI capabilities for the Information Extractor, enabling advanced natural language understanding and data parsing.
  4. Structured Output: A Structured Output Parser node (from Langchain) ensures that the information extracted by the AI is formatted into a consistent, structured output, likely JSON.
  5. Conditional Logic: A Switch node allows for conditional branching based on the extracted information, enabling different actions depending on the data's content or validity.
  6. Data Storage (Supabase): The workflow is configured to interact with Supabase, indicating that the extracted company intelligence is intended to be stored in a Supabase database.
  7. File Conversion: A Convert to File node suggests that the extracted data or other intermediate results might be converted into various file formats (e.g., CSV, JSON) for further processing or storage.
  8. HTTP Request Tool: An HTTP Request Tool (from Langchain) is integrated, potentially allowing the AI agent to make external web requests as part of its information gathering or validation process.
  9. Google Drive Integration: A Google Drive node is present, suggesting that files, reports, or other data might be stored or retrieved from Google Drive.
  10. AI Agent: An AI Agent node (from Langchain) indicates the presence of a more complex, conversational or "plan and execute" AI component, which could orchestrate the information gathering and extraction process.
  11. No Operation: A No Operation, do nothing node is included, which typically acts as a placeholder or a point where the workflow can end without performing any further action.

Note: The workflow JSON primarily defines the nodes and their connections, but the specific configuration (e.g., what data the form collects, what information the AI extracts, which Supabase table is used) would be set within each node's parameters.

Prerequisites/Requirements

  • n8n Instance: A running n8n instance to import and execute the workflow.
  • Mistral Cloud Account: Credentials for the Mistral Cloud Chat Model (API Key).
  • Supabase Account: Credentials and a configured database for storing extracted data.
  • Google Drive Account: Credentials for accessing Google Drive (if used for storage).
  • n8n Form Trigger: The form needs to be embedded or linked to trigger the workflow.

Setup/Usage

  1. Import the workflow: Download the provided JSON and import it into your n8n instance.
  2. Configure Credentials:
    • Set up your Mistral Cloud Chat Model credentials (API Key).
    • Configure your Supabase credentials and specify the database/table where you want to store the company intelligence.
    • (Optional) Configure your Google Drive credentials if you intend to use that node.
  3. Configure the n8n Form Trigger:
    • Define the fields your form will collect. These fields will be the input for the AI extraction.
    • Share or embed the generated form URL as needed.
  4. Configure the Information Extractor:
    • Specify the schema or the type of information you want the AI to extract (e.g., company name, website, industry, contact details, funding rounds, etc.).
    • Map the input data from the form trigger to the extractor.
  5. Configure the Structured Output Parser:
    • Ensure it matches the expected output format from the Information Extractor.
  6. Configure the Switch node:
    • Define the conditions for branching based on the extracted data (e.g., if a company name is found, if certain keywords are present).
  7. Activate the workflow: Once configured, activate the workflow to start processing form submissions.

Related Templates

Track competitor SEO keywords with Decodo + GPT-4.1-mini + Google Sheets

This workflow automates competitor keyword research using OpenAI LLM and Decodo for intelligent web scraping. Who this is for SEO specialists, content strategists, and growth marketers who want to automate keyword research and competitive intelligence. Marketing analysts managing multiple clients or websites who need consistent SEO tracking without manual data pulls. Agencies or automation engineers using Google Sheets as an SEO data dashboard for keyword monitoring and reporting. What problem this workflow solves Tracking competitor keywords manually is slow and inconsistent. Most SEO tools provide limited API access or lack contextual keyword analysis. This workflow solves that by: Automatically scraping any competitor’s webpage with Decodo. Using OpenAI GPT-4.1-mini to interpret keyword intent, density, and semantic focus. Storing structured keyword insights directly in Google Sheets for ongoing tracking and trend analysis. What this workflow does Trigger — Manually start the workflow or schedule it to run periodically. Input Setup — Define the website URL and target country (e.g., https://dev.to, france). Data Scraping (Decodo) — Fetch competitor web content and metadata. Keyword Analysis (OpenAI GPT-4.1-mini) Extract primary and secondary keywords. Identify focus topics and semantic entities. Generate a keyword density summary and SEO strength score. Recommend optimization and internal linking opportunities. Data Structuring — Clean and convert GPT output into JSON format. Data Storage (Google Sheets) — Append structured keyword data to a Google Sheet for long-term tracking. Setup Prerequisites If you are new to Decode, please signup on this link visit.decodo.com n8n account with workflow editor access Decodo API credentials OpenAI API key Google Sheets account connected via OAuth2 Make sure to install the Decodo Community node. Create a Google Sheet Add columns for: primarykeywords, seostrengthscore, keyworddensity_summary, etc. Share with your n8n Google account. Connect Credentials Add credentials for: Decodo API credentials - You need to register, login and obtain the Basic Authentication Token via Decodo Dashboard OpenAI API (for GPT-4o-mini) Google Sheets OAuth2 Configure Input Fields Edit the “Set Input Fields” node to set your target site and region. Run the Workflow Click Execute Workflow in n8n. View structured results in your connected Google Sheet. How to customize this workflow Track Multiple Competitors → Use a Google Sheet or CSV list of URLs; loop through them using the Split In Batches node. Add Language Detection → Add a Gemini or GPT node before keyword analysis to detect content language and adjust prompts. Enhance the SEO Report → Expand the GPT prompt to include backlink insights, metadata optimization, or readability checks. Integrate Visualization → Connect your Google Sheet to Looker Studio for SEO performance dashboards. Schedule Auto-Runs → Use the Cron Node to run weekly or monthly for competitor keyword refreshes. Summary This workflow automates competitor keyword research using: Decodo for intelligent web scraping OpenAI GPT-4.1-mini for keyword and SEO analysis Google Sheets for live tracking and reporting It’s a complete AI-powered SEO intelligence pipeline ideal for teams that want actionable insights on keyword gaps, optimization opportunities, and content focus trends, without relying on expensive SEO SaaS tools.

Ranjan DailataBy Ranjan Dailata
161

Generate song lyrics and music from text prompts using OpenAI and Fal.ai Minimax

Spark your creativity instantly in any chat—turn a simple prompt like "heartbreak ballad" into original, full-length lyrics and a professional AI-generated music track, all without leaving your conversation. 📋 What This Template Does This chat-triggered workflow harnesses AI to generate detailed, genre-matched song lyrics (at least 600 characters) from user messages, then queues them for music synthesis via Fal.ai's minimax-music model. It polls asynchronously until the track is ready, delivering lyrics and audio URL back in chat. Crafts original, structured lyrics with verses, choruses, and bridges using OpenAI Submits to Fal.ai for melody, instrumentation, and vocals aligned to the style Handles long-running generations with smart looping and status checks Returns complete song package (lyrics + audio link) for seamless sharing 🔧 Prerequisites n8n account (self-hosted or cloud with chat integration enabled) OpenAI account with API access for GPT models Fal.ai account for AI music generation 🔑 Required Credentials OpenAI API Setup Go to platform.openai.com → API keys (sidebar) Click "Create new secret key" → Name it (e.g., "n8n Songwriter") Copy the key and add to n8n as "OpenAI API" credential type Test by sending a simple chat completion request Fal.ai HTTP Header Auth Setup Sign up at fal.ai → Dashboard → API Keys Generate a new API key → Copy it In n8n, create "HTTP Header Auth" credential: Name="Fal.ai", Header Name="Authorization", Header Value="Key [Your API Key]" Test with a simple GET to their queue endpoint (e.g., /status) ⚙️ Configuration Steps Import the workflow JSON into your n8n instance Assign OpenAI API credentials to the "OpenAI Chat Model" node Assign Fal.ai HTTP Header Auth to the "Generate Music Track", "Check Generation Status", and "Fetch Final Result" nodes Activate the workflow—chat trigger will appear in your n8n chat interface Test by messaging: "Create an upbeat pop song about road trips" 🎯 Use Cases Content Creators: YouTubers generating custom jingles for videos on the fly, streamlining production from idea to audio export Educators: Music teachers using chat prompts to create era-specific folk tunes for classroom discussions, fostering interactive learning Gift Personalization: Friends crafting anniversary R&B tracks from shared memories via quick chats, delivering emotional audio surprises Artist Brainstorming: Songwriters prototyping hip-hop beats in real-time during sessions, accelerating collaboration and iteration ⚠️ Troubleshooting Invalid JSON from AI Agent: Ensure the system prompt stresses valid JSON; test the agent standalone with a sample query Music Generation Fails (401/403): Verify Fal.ai API key has minimax-music access; check usage quotas in dashboard Status Polling Loops Indefinitely: Bump wait time to 45-60s for complex tracks; inspect fal.ai queue logs for bottlenecks Lyrics Under 600 Characters: Tweak agent prompt to enforce fuller structures like [V1][C][V2][B][C]; verify output length in executions

Daniel NkenchoBy Daniel Nkencho
601

Automate invoice processing with OCR, GPT-4 & Salesforce opportunity creation

PDF Invoice Extractor (AI) End-to-end pipeline: Watch Drive ➜ Download PDF ➜ OCR text ➜ AI normalize to JSON ➜ Upsert Buyer (Account) ➜ Create Opportunity ➜ Map Products ➜ Create OLI via Composite API ➜ Archive to OneDrive. --- Node by node (what it does & key setup) 1) Google Drive Trigger Purpose: Fire when a new file appears in a specific Google Drive folder. Key settings: Event: fileCreated Folder ID: google drive folder id Polling: everyMinute Creds: googleDriveOAuth2Api Output: Metadata { id, name, ... } for the new file. --- 2) Download File From Google Purpose: Get the file binary for processing and archiving. Key settings: Operation: download File ID: ={{ $json.id }} Creds: googleDriveOAuth2Api Output: Binary (default key: data) and original metadata. --- 3) Extract from File Purpose: Extract text from PDF (OCR as needed) for AI parsing. Key settings: Operation: pdf OCR: enable for scanned PDFs (in options) Output: JSON with OCR text at {{ $json.text }}. --- 4) Message a model (AI JSON Extractor) Purpose: Convert OCR text into strict normalized JSON array (invoice schema). Key settings: Node: @n8n/n8n-nodes-langchain.openAi Model: gpt-4.1 (or gpt-4.1-mini) Message role: system (the strict prompt; references {{ $json.text }}) jsonOutput: true Creds: openAiApi Output (per item): $.message.content → the parsed JSON (ensure it’s an array). --- 5) Create or update an account (Salesforce) Purpose: Upsert Buyer as Account using an external ID. Key settings: Resource: account Operation: upsert External Id Field: taxid_c External Id Value: ={{ $json.message.content.buyer.tax_id }} Name: ={{ $json.message.content.buyer.name }} Creds: salesforceOAuth2Api Output: Account record (captures Id) for downstream Opportunity. --- 6) Create an opportunity (Salesforce) Purpose: Create Opportunity linked to the Buyer (Account). Key settings: Resource: opportunity Name: ={{ $('Message a model').item.json.message.content.invoice.code }} Close Date: ={{ $('Message a model').item.json.message.content.invoice.issue_date }} Stage: Closed Won Amount: ={{ $('Message a model').item.json.message.content.summary.grand_total }} AccountId: ={{ $json.id }} (from Upsert Account output) Creds: salesforceOAuth2Api Output: Opportunity Id for OLI creation. --- 7) Build SOQL (Code / JS) Purpose: Collect unique product codes from AI JSON and build a SOQL query for PricebookEntry by Pricebook2Id. Key settings: pricebook2Id (hardcoded in script): e.g., 01sxxxxxxxxxxxxxxx Source lines: $('Message a model').first().json.message.content.products Output: { soql, codes } --- 8) Query PricebookEntries (Salesforce) Purpose: Fetch PricebookEntry.Id for each Product2.ProductCode. Key settings: Resource: search Query: ={{ $json.soql }} Creds: salesforceOAuth2Api Output: Items with Id, Product2.ProductCode (used for mapping). --- 9) Code in JavaScript (Build OLI payloads) Purpose: Join lines with PBE results and Opportunity Id ➜ build OpportunityLineItem payloads. Inputs: OpportunityId: ={{ $('Create an opportunity').first().json.id }} Lines: ={{ $('Message a model').first().json.message.content.products }} PBE rows: from previous node items Output: { body: { allOrNone:false, records:[{ OpportunityLineItem... }] } } Notes: Converts discount_total ➜ per-unit if needed (currently commented for standard pricing). Throws on missing PBE mapping or empty lines. --- 10) Create Opportunity Line Items (HTTP Request) Purpose: Bulk create OLIs via Salesforce Composite API. Key settings: Method: POST URL: https://<your-instance>.my.salesforce.com/services/data/v65.0/composite/sobjects Auth: salesforceOAuth2Api (predefined credential) Body (JSON): ={{ $json.body }} Output: Composite API results (per-record statuses). --- 11) Update File to One Drive Purpose: Archive the original PDF in OneDrive. Key settings: Operation: upload File Name: ={{ $json.name }} Parent Folder ID: onedrive folder id Binary Data: true (from the Download node) Creds: microsoftOneDriveOAuth2Api Output: Uploaded file metadata. --- Data flow (wiring) Google Drive Trigger → Download File From Google Download File From Google → Extract from File → Update File to One Drive Extract from File → Message a model Message a model → Create or update an account Create or update an account → Create an opportunity Create an opportunity → Build SOQL Build SOQL → Query PricebookEntries Query PricebookEntries → Code in JavaScript Code in JavaScript → Create Opportunity Line Items --- Quick setup checklist 🔐 Credentials: Connect Google Drive, OneDrive, Salesforce, OpenAI. 📂 IDs: Drive Folder ID (watch) OneDrive Parent Folder ID (archive) Salesforce Pricebook2Id (in the JS SOQL builder) 🧠 AI Prompt: Use the strict system prompt; jsonOutput = true. 🧾 Field mappings: Buyer tax id/name → Account upsert fields Invoice code/date/amount → Opportunity fields Product name must equal your Product2.ProductCode in SF. ✅ Test: Drop a sample PDF → verify: AI returns array JSON only Account/Opportunity created OLI records created PDF archived to OneDrive --- Notes & best practices If PDFs are scans, enable OCR in Extract from File. If AI returns non-JSON, keep “Return only a JSON array” as the last line of the prompt and keep jsonOutput enabled. Consider adding validation on parsing.warnings to gate Salesforce writes. For discounts/taxes in OLI: Standard OLI fields don’t support per-line discount amounts directly; model them in UnitPrice or custom fields. Replace the Composite API URL with your org’s domain or use the Salesforce node’s Bulk Upsert for simplicity.

Le NguyenBy Le Nguyen
942