HumationHuman + Automation
Back to Catalog

Create AI-ready vector datasets from web content with Claude, Ollama & Qdrant

scrapeless officialscrapeless official
1353 views
2/3/2026
Beginner FriendlyWebhookData TransformationETLBatch Processing
Official Page

AI-Powered Web Data Pipeline with n8n

f70ca7d4da7544ce92333c57c5e62954.png

How It Works

This n8n workflow builds an AI-powered web data pipeline that automates the entire process of:

  • Extraction
  • Structuring
  • Vectorization
  • Storage

It integrates multiple advanced tools to transform messy web pages into clean, searchable vector databases.

Integrated Tools

  • Scrapeless
    Bypasses JavaScript-heavy websites and anti-bot protections to reliably extract HTML content.

  • Claude AI
    Uses LLMs to analyze unstructured HTML and generate clean, structured JSON data.

  • Ollama Embeddings
    Generates local vector embeddings from structured text using the all-minilm model.

  • Qdrant Vector DB
    Stores semantic vector data for fast and meaningful search capabilities.

  • Webhook Notifications
    Sends real-time updates when workflows complete or errors occur.

From messy webpages to structured vector data — this pipeline is perfect for building intelligent agents, knowledge bases, or research automation tools.

Setup Steps

1. Install n8n

> Requires Node.js v18 / v20 / v22

npm install -g n8n
n8n

After installation, access the n8n interface via:

URL: http://localhost:5678

2. Set Up Scrapeless

  1. Register at: Scrapeless
  2. Copy your API token
  3. Paste the token into the HTTP Request node labeled "Scrapeless Web Request"

3. Set Up Claude API (Anthropic)

  1. Sign up at Anthropic Console
  2. Generate your Claude API key
  3. Add the API key to the following nodes:
    • Claude Extractor
    • AI Data Checker
    • Claude AI Agent

4. Install and Run Ollama

macOS

brew install ollama

Linux

curl -fsSL https://ollama.com/install.sh | sh

Windows Download the installer from: https://ollama.com

Start Ollama Server

ollama serve

Pull Embedding Model

ollama pull all-minilm

5. Install Qdrant (via Docker)

docker pull qdrant/qdrant

docker run -d \
  --name qdrant-server \
  -p 6333:6333 -p 6334:6334 \
  -v $(pwd)/qdrant_storage:/qdrant/storage \
  qdrant/qdrant

Test if Qdrant is running:

curl http://localhost:6333/healthz

6. Configure the n8n Workflow

  • Modify the Trigger (Manual or Scheduled)

  • Input your Target URLs and Collection Name in the designated nodes

  • Paste all required API Tokens / Keys into their corresponding nodes

  • Ensure your Qdrant and Ollama services are running

Ideal Use Cases

  • Custom AI Chatbots

  • Private Search Engines

  • Research Tools

  • Internal Knowledge Bases

  • Content Monitoring Pipelines

n8n Workflow: Create AI-Ready Vector Datasets from Web Content

This n8n workflow provides a foundational structure for processing web content to prepare it for AI applications, specifically focusing on creating vector datasets. It demonstrates how to initiate a process, make an HTTP request, and apply conditional logic based on the response.

What it does

This workflow outlines a process that can be extended to:

  1. Manually Trigger Execution: Starts the workflow on demand, allowing for manual initiation of the data processing.
  2. Make an HTTP Request: Sends a request to a specified URL. This is typically used to fetch web content, interact with an API, or initiate an external process.
  3. Apply Conditional Logic: Evaluates the outcome of the HTTP request. This allows for branching the workflow based on whether the request was successful or met specific criteria.
  4. Edit Fields (Set): If the HTTP request meets the defined condition, this node prepares or transforms the data for further processing, such as extracting relevant information or structuring it for vectorization.
  5. Execute Custom Code: If the HTTP request does not meet the defined condition, this node allows for running custom JavaScript code. This could be used for error handling, logging, or alternative data processing paths.

Prerequisites/Requirements

  • n8n Instance: A running instance of n8n to import and execute the workflow.
  • Target API/URL: An endpoint or web page that the HTTP Request node will interact with.

Setup/Usage

  1. Import the workflow: Copy the provided JSON and import it into your n8n instance.
  2. Configure the "HTTP Request" node:
    • Set the URL to the web content source or API endpoint you wish to interact with.
    • Configure any necessary Headers, Query Parameters, or Body for your request.
  3. Configure the "If" node:
    • Define the Conditions to evaluate the response from the "HTTP Request" node. For example, you might check for a specific HTTP status code (e.g., 200 for success) or a particular value in the response body.
  4. Configure the "Edit Fields (Set)" node (True branch):
    • If the "If" node evaluates to True, this node will execute. Configure it to extract, rename, or transform data from the HTTP response as needed for your AI-ready dataset.
  5. Configure the "Code" node (False branch):
    • If the "If" node evaluates to False, this node will execute. Add custom JavaScript code here for error handling, logging, or alternative processing.
  6. Extend the workflow: This workflow provides a starting point. You can connect additional nodes after the "Edit Fields (Set)" node to:
    • Extract Text: Use a "Cheerio" or "HTML Extract" node to parse web content.
    • Generate Embeddings: Integrate with an AI model (e.g., OpenAI, Cohere, local Ollama) to create vector embeddings from the extracted text.
    • Store Vectors: Connect to a vector database (e.g., Qdrant, Pinecone, Weaviate) to store the generated vectors and associated metadata.
    • Notifications: Send notifications (e.g., Slack, Email) about the processing status or errors.
  7. Execute the workflow: Click the "Execute Workflow" button on the "Manual Trigger" node to run the workflow.

Related Templates

Automate Dutch Public Procurement Data Collection with TenderNed

TenderNed Public Procurement What This Workflow Does This workflow automates the collection of public procurement data from TenderNed (the official Dutch tender platform). It: Fetches the latest tender publications from the TenderNed API Retrieves detailed information in both XML and JSON formats for each tender Parses and extracts key information like organization names, titles, descriptions, and reference numbers Filters results based on your custom criteria Stores the data in a database for easy querying and analysis Setup Instructions This template comes with sticky notes providing step-by-step instructions in Dutch and various query options you can customize. Prerequisites TenderNed API Access - Register at TenderNed for API credentials Configuration Steps Set up TenderNed credentials: Add HTTP Basic Auth credentials with your TenderNed API username and password Apply these credentials to the three HTTP Request nodes: "Tenderned Publicaties" "Haal XML Details" "Haal JSON Details" Customize filters: Modify the "Filter op ..." node to match your specific requirements Examples: specific organizations, contract values, regions, etc. How It Works Step 1: Trigger The workflow can be triggered either manually for testing or automatically on a daily schedule. Step 2: Fetch Publications Makes an API call to TenderNed to retrieve a list of recent publications (up to 100 per request). Step 3: Process & Split Extracts the tender array from the response and splits it into individual items for processing. Step 4: Fetch Details For each tender, the workflow makes two parallel API calls: XML endpoint - Retrieves the complete tender documentation in XML format JSON endpoint - Fetches metadata including reference numbers and keywords Step 5: Parse & Merge Parses the XML data and merges it with the JSON metadata and batch information into a single data structure. Step 6: Extract Fields Maps the raw API data to clean, structured fields including: Publication ID and date Organization name Tender title and description Reference numbers (kenmerk, TED number) Step 7: Filter Applies your custom filter criteria to focus on relevant tenders only. Step 8: Store Inserts the processed data into your database for storage and future analysis. Customization Tips Modify API Parameters In the "Tenderned Publicaties" node, you can adjust: offset: Starting position for pagination size: Number of results per request (max 100) Add query parameters for date ranges, status filters, etc. Add More Fields Extend the "Splits Alle Velden" node to extract additional fields from the XML/JSON data, such as: Contract value estimates Deadline dates CPV codes (procurement classification) Contact information Integrate Notifications Add a Slack, Email, or Discord node after the filter to get notified about new matching tenders. Incremental Updates Modify the workflow to only fetch new tenders by: Storing the last execution timestamp Adding date filters to the API query Only processing publications newer than the last run Troubleshooting No data returned? Verify your TenderNed API credentials are correct Check that you have setup youre filter proper Need help setting this up or interested in a complete tender analysis solution? Get in touch 🔗 LinkedIn – Wessel Bulte

Wessel BulteBy Wessel Bulte
247

AI-powered code review with linting, red-marked corrections in Google Sheets & Slack

Advanced Code Review Automation (AI + Lint + Slack) Who’s it for For software engineers, QA teams, and tech leads who want to automate intelligent code reviews with both AI-driven suggestions and rule-based linting — all managed in Google Sheets with instant Slack summaries. How it works This workflow performs a two-layer review system: Lint Check: Runs a lightweight static analysis to find common issues (e.g., use of var, console.log, unbalanced braces). AI Review: Sends valid code to Gemini AI, which provides human-like review feedback with severity classification (Critical, Major, Minor) and visual highlights (red/orange tags). Formatter: Combines lint and AI results, calculating an overall score (0–10). Aggregator: Summarizes results for quick comparison. Google Sheets Writer: Appends results to your review log. Slack Notification: Posts a concise summary (e.g., number of issues and average score) to your team’s channel. How to set up Connect Google Sheets and Slack credentials in n8n. Replace placeholders (<YOURSPREADSHEETID>, <YOURSHEETGIDORNAME>, <YOURSLACKCHANNEL_ID>). Adjust the AI review prompt or lint rules as needed. Activate the workflow — reviews will start automatically whenever new code is added to the sheet. Requirements Google Sheets and Slack integrations enabled A configured AI node (Gemini, OpenAI, or compatible) Proper permissions to write to your target Google Sheet How to customize Add more linting rules (naming conventions, spacing, forbidden APIs) Extend the AI prompt for project-specific guidelines Customize the Slack message formatting Export analytics to a dashboard (e.g., Notion or Data Studio) Why it’s valuable This workflow brings realistic, team-oriented AI-assisted code review to n8n — combining the speed of automated linting with the nuance of human-style feedback. It saves time, improves code quality, and keeps your team’s review history transparent and centralized.

higashiyama By higashiyama
90

🎓 How to transform unstructured email data into structured format with AI agent

This workflow automates the process of extracting structured, usable information from unstructured email messages across multiple platforms. It connects directly to Gmail, Outlook, and IMAP accounts, retrieves incoming emails, and sends their content to an AI-powered parsing agent built on OpenAI GPT models. The AI agent analyzes each email, identifies relevant details, and returns a clean JSON structure containing key fields: From – sender’s email address To – recipient’s email address Subject – email subject line Summary – short AI-generated summary of the email body The extracted information is then automatically inserted into an n8n Data Table, creating a structured database of email metadata and summaries ready for indexing, reporting, or integration with other tools. --- Key Benefits ✅ Full Automation: Eliminates manual reading and data entry from incoming emails. ✅ Multi-Source Integration: Handles data from different email providers seamlessly. ✅ AI-Driven Accuracy: Uses advanced language models to interpret complex or unformatted content. ✅ Structured Storage: Creates a standardized, query-ready dataset from previously unstructured text. ✅ Time Efficiency: Processes emails in real time, improving productivity and response speed. *✅ Scalability: Easily extendable to handle additional sources or extract more data fields. --- How it works This workflow automates the transformation of unstructured email data into a structured, queryable format. It operates through a series of connected steps: Email Triggering: The workflow is initiated by one of three different email triggers (Gmail, Microsoft Outlook, or a generic IMAP account), which constantly monitor for new incoming emails. AI-Powered Parsing & Structuring: When a new email is detected, its raw, unstructured content is passed to a central "Parsing Agent." This agent uses a specified OpenAI language model to intelligently analyze the email text. Data Extraction & Standardization: Following a predefined system prompt, the AI agent extracts key information from the email, such as the sender, recipient, subject, and a generated summary. It then forces the output into a strict JSON structure using a "Structured Output Parser" node, ensuring data consistency. Data Storage: Finally, the clean, structured data (the from, to, subject, and summarize fields) is inserted as a new row into a specified n8n Data Table, creating a searchable and reportable database of email information. --- Set up steps To implement this workflow, follow these configuration steps: Prepare the Data Table: Create a new Data Table within n8n. Define the columns with the following names and string type: From, To, Subject, and Summary. Configure Email Credentials: Set up the credential connections for the email services you wish to use (Gmail OAuth2, Microsoft Outlook OAuth2, and/or IMAP). Ensure the accounts have the necessary permissions to read emails. Configure AI Model Credentials: Set up the OpenAI API credential with a valid API key. The workflow is configured to use the model, but this can be changed in the respective nodes if needed. Connect the Nodes: The workflow canvas is already correctly wired. Visually confirm that the email triggers are connected to the "Parsing Agent," which is connected to the "Insert row" (Data Table) node. Also, ensure the "OpenAI Chat Model" and "Structured Output Parser" are connected to the "Parsing Agent" as its AI model and output parser, respectively. Activate the Workflow: Save the workflow and toggle the "Active" switch to ON. The triggers will begin polling for new emails according to their schedule (e.g., every minute), and the automation will start processing incoming messages. --- Need help customizing? Contact me for consulting and support or add me on Linkedin.

DavideBy Davide
1616