n8n Workflow: Index Legal Documents for Hybrid Search with Qdrant and OpenAI

This n8n workflow demonstrates how to process legal documents, generate embeddings using OpenAI, and index them into Qdrant for hybrid search capabilities. It's designed to prepare a dataset of legal documents for efficient semantic and keyword-based retrieval.

What it does

This workflow automates the following steps:

1. Manual Trigger: Starts the workflow manually, allowing you to initiate the indexing process on demand.
2. Edit Fields (Set): This node is likely used to define or modify input data, such as the source of legal documents or parameters for processing. (The provided JSON doesn't specify the exact fields, but this is its typical use).
3. Loop Over Items (Split in Batches): Iterates over a collection of legal documents or data items, processing them in batches to manage API limits or resource usage.
4. HTTP Request (OpenAI Embeddings): For each document (or batch of documents), it sends a request to the OpenAI API to generate vector embeddings. These embeddings capture the semantic meaning of the document content.
5. HTTP Request (Qdrant Upsert): After obtaining the embeddings, the workflow sends another HTTP request to the Qdrant vector database to upsert (insert or update) the document. This includes the original document content (payload) and its generated vector embedding.
6. Merge: Combines the output from the batch processing, likely aggregating results or ensuring all items have been processed before proceeding.
7. If: This node introduces conditional logic. It likely checks the success of the Qdrant upsert operation or other conditions to determine the next steps.
8. Limit: This node is present but not connected in the provided JSON, suggesting it might be a placeholder or an unused component. If connected, it would restrict the number of items passing through.
9. Split Out: This node is also present but not connected, indicating it might be an unused component. If connected, it would typically split an array of items into individual items.
10. Aggregate: This node is present but not connected. If connected, it would typically combine items into a single collection or perform aggregation operations.
11. Sticky Note: Provides a place to add comments or documentation directly within the workflow canvas.

Prerequisites/Requirements

To use this workflow, you will need:

• OpenAI API Key: For generating document embeddings.
• Qdrant Instance: A running Qdrant vector database instance where the documents will be indexed. This could be self-hosted or a cloud service.
• n8n Instance: A running n8n instance to host and execute the workflow.

Setup/Usage

1. Import the Workflow: Download the JSON provided and import it into your n8n instance.
2. Configure Credentials:
   • OpenAI: Configure an HTTP Request node (likely node ID 19, if it's the OpenAI one) with your OpenAI API key. This will typically involve setting an Authorization header with Bearer YOUR_OPENAI_API_KEY.
   • Qdrant: Configure the HTTP Request node responsible for Qdrant upsert (likely node ID 19 as well, or another HTTP Request node if there are two distinct ones) with the URL of your Qdrant instance and any necessary API keys or authentication headers.
3. Define Input Data: Modify the "Edit Fields (Set)" node (node ID 38) to provide the legal documents you wish to index. This could be an array of objects, where each object represents a document with its content and any metadata.
4. Activate and Execute:
   • Ensure the workflow is active.
   • Click "Execute Workflow" on the "Manual Trigger" node (node ID 838) to start the indexing process.

This workflow provides a robust foundation for building a hybrid search solution for legal documents, leveraging the power of large language models for semantic understanding and vector databases for efficient retrieval.