n8n Workflow: AI-Ready Vector Dataset Creation for LLMs with Bright Data, Gemini, and Pinecone

This n8n workflow demonstrates how to create AI-ready vector datasets for Large Language Models (LLMs) by extracting information from web data (simulated via an HTTP Request), processing it with Google Gemini for embeddings, and storing it in a Pinecone vector database.

What it does

This workflow automates the process of transforming raw data into structured, vectorized information suitable for LLM applications. Here's a step-by-step breakdown:

1. Manual Trigger: The workflow is initiated manually, allowing for on-demand execution.
2. HTTP Request (Simulated Data Source): This node simulates fetching data from a source (e.g., a web scraper like Bright Data, though not explicitly configured in this JSON). It acts as the initial input for the data processing.
3. Edit Fields (Set): This node is typically used to transform or prepare the data received from the previous step, ensuring it's in the correct format for subsequent AI processing.
4. Default Data Loader: This LangChain node loads the processed data into a document format, making it ready for text splitting and embedding.
5. Recursive Character Text Splitter: This LangChain node breaks down the loaded documents into smaller, manageable chunks (text splits). This is crucial for handling large documents and optimizing embedding generation.
6. Embeddings Google Gemini: This LangChain node uses the Google Gemini model to generate vector embeddings for each text chunk. These embeddings capture the semantic meaning of the text.
7. Pinecone Vector Store: The generated vector embeddings are then stored in a Pinecone vector database. This node handles the connection and indexing of the vectors, making them searchable for LLM applications.
8. Information Extractor: This LangChain node is designed to extract structured information from the processed data. It can be used to identify entities, relationships, or specific data points.
9. Basic LLM Chain: This LangChain node represents a fundamental LLM interaction, likely used for further processing or querying the extracted information.
10. Structured Output Parser: This LangChain node parses the output from an LLM or other AI components into a structured format, such as JSON, for easier consumption by subsequent nodes or applications.
11. AI Agent: This LangChain node represents an AI agent that can perform more complex, multi-step tasks, potentially leveraging the vector store for retrieval-augmented generation (RAG).
12. Google Gemini Chat Model: This LangChain node provides a chat interface to the Google Gemini LLM, allowing for conversational interactions or more advanced text generation.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running n8n instance (self-hosted or cloud).
• Pinecone Account: An account with Pinecone to store your vector embeddings. You'll need API keys and an index configured.
• Google Cloud Account / Gemini API Access: Access to Google Gemini for generating embeddings and using the chat model. This typically involves setting up a Google Cloud project and enabling the Gemini API.
• Bright Data (Optional, but implied by directory name): While not explicitly configured in the JSON, the directory name suggests an integration with Bright Data for web scraping. If you intend to use Bright Data as your primary data source, you'll need an account and a configured scraper. The "HTTP Request" node would then be configured to call your Bright Data API.

Setup/Usage

1. Import the workflow: Download the provided JSON and import it into your n8n instance.
2. Configure Credentials:
   • Pinecone Vector Store: Configure your Pinecone credentials (API Key, Environment, Index Name) within this node.
   • Embeddings Google Gemini: Configure your Google Gemini API key or credentials for embedding generation.
   • Google Gemini Chat Model: Configure your Google Gemini API key or credentials for the chat model.
3. Review and Adjust HTTP Request: The "HTTP Request" node is currently a generic placeholder. You will need to configure it to fetch data from your desired source. If using Bright Data, point it to your Bright Data API endpoint.
4. Customize Edit Fields (Set): Adjust the "Edit Fields" node to transform the incoming data into the structure required for your specific use case.
5. Configure LangChain Nodes: Review the settings for the "Default Data Loader", "Recursive Character Text Splitter", "Information Extractor", "Basic LLM Chain", "Structured Output Parser", "AI Agent", and "Google Gemini Chat Model" nodes to match your specific data processing and LLM interaction requirements.
6. Execute the Workflow: Click the "Execute workflow" button on the "Manual Trigger" node to run the workflow. Observe the output of each node to ensure data is processed as expected.