Automate Web Research with GPT-4/Claude & Apify for Content Analysis and Insights

This n8n workflow automates the process of extracting content from web pages, analyzing it with advanced AI models (GPT-4 or Claude), and generating structured insights. It's designed to streamline web research, content summarization, and data extraction for various applications.

What it does

This workflow performs the following key steps:

1. Triggers Manually or via External Workflow: The workflow can be executed manually or by another n8n workflow.
2. Initial Data Setup: Sets up an initial item with a placeholder text field, which is intended to be replaced by actual web content.
3. Fetches Web Content (Placeholder): An HTTP Request node is present, likely intended to fetch content from a URL. In its current state, it's a placeholder.
4. Extracts & Processes Content:
   • Code Node: A Code node is used to define a structured output schema (using Zod) for the AI to follow. This schema expects a title, summary, keywords, and main_points from the analyzed content.
   • Character Text Splitter: Splits the input text into manageable chunks for processing by the AI.
   • Default Data Loader: Loads the split text chunks as documents.
   • Qdrant Vector Store: Stores the processed documents as vectors in a Qdrant vector database, enabling efficient semantic search and retrieval.
   • Embeddings Ollama: Generates embeddings for the text chunks using the Ollama embedding model, which are then used by the Qdrant Vector Store.
5. Analyzes Content with AI Agent:
   • AI Agent: Utilizes an AI Agent (LangChain) to perform complex reasoning and content analysis based on the provided text and the defined output schema.
   • OpenAI Chat Model / Anthropic Chat Model: The AI Agent can leverage either an OpenAI (GPT-4) or Anthropic (Claude) chat model for its analytical tasks. The workflow includes both, allowing for flexibility.
   • Structured Output Parser: Parses the AI agent's response to ensure it adheres to the predefined JSON schema, extracting the title, summary, keywords, and main_points.
6. Conditional AI Model Selection: An "If" node is included, suggesting a conditional logic to choose between different AI models or processing paths, though the specific condition is not defined in the provided JSON.
7. Merges Results: A Merge node is present, likely to combine results from different branches or steps of the workflow.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running instance of n8n.
• OpenAI API Key: For using the OpenAI Chat Model (e.g., GPT-4).
• Anthropic API Key: For using the Anthropic Chat Model (e.g., Claude).
• Qdrant Vector Database: Access to a Qdrant instance for vector storage.
• Ollama Embeddings: An Ollama instance running with the desired embedding model for text embeddings.
• Apify (Implied): While not explicitly present in the provided JSON, the directory name suggests an integration with Apify for web scraping. If used, an Apify account and API key would be required.

Setup/Usage

1. Import the Workflow: Download the workflow JSON and import it into your n8n instance.
2. Configure Credentials:
   • Set up credentials for your OpenAI and/or Anthropic API keys.
   • Configure the Qdrant Vector Store node with your Qdrant instance details (host, API key).
   • Configure the Embeddings Ollama node with your Ollama instance details.
3. Customize Web Content Source:
   • The HTTP Request node (ID 19) is currently a placeholder. Replace it with a node that fetches the actual web content you wish to analyze. This could be an Apify node, another HTTP Request configured to scrape a specific URL, or a Webhook to receive content.
4. Adjust AI Agent Prompt and Schema:
   • Review the Code node (ID 834) to understand the structured output schema. Modify it if you need different fields from the AI analysis.
   • Adjust the AI Agent node (ID 1119) to refine the prompt and instructions given to the AI for content analysis.
5. Define Conditional Logic (Optional):
   • If you intend to use the If node (ID 20) for conditional routing (e.g., choosing between OpenAI and Anthropic based on input), configure its conditions accordingly.
6. Activate and Execute: Once configured, activate the workflow. You can execute it manually or trigger it via an external source as per your setup.

This workflow provides a powerful foundation for automated web research and content analysis, leveraging the strengths of large language models and vector databases.