n8n Workflow: FTP JSON to Qdrant Vector Database Embedding Pipeline

This n8n workflow automates the process of ingesting JSON files from an FTP server, processing their content, generating embeddings using OpenAI, and storing these embeddings in a Qdrant vector database. This is ideal for scenarios where you need to keep a vector database updated with information from structured data files stored on an FTP server.

What it does

This workflow performs the following steps:

1. Manual Trigger: The workflow starts when manually executed.
2. FTP File Listing: Connects to an FTP server and retrieves a list of files.
3. Loop Over Items: Iterates through each file found on the FTP server.
4. Default Data Loader: Loads the content of each file. Assuming the files are JSON, this node will prepare the data for further processing.
5. Character Text Splitter: Splits the loaded text content into smaller, manageable chunks. This is crucial for handling large documents and optimizing embedding generation.
6. Embeddings OpenAI: Generates vector embeddings for each text chunk using the OpenAI API.
7. Qdrant Vector Store: Stores the generated embeddings and their associated metadata into a Qdrant vector database.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running n8n instance.
• FTP Server Access: Credentials (host, port, username, password) for an FTP server containing the JSON files you wish to process.
• OpenAI API Key: An API key for OpenAI to generate text embeddings.
• Qdrant Instance: Access to a Qdrant vector database instance (host, port, API key if applicable).

Setup/Usage

1. Import the workflow:
   • Copy the provided JSON code.
   • In your n8n instance, go to "Workflows" and click "New".
   • Click the three dots (...) in the top right corner and select "Import from JSON".
   • Paste the JSON code and click "Import".
2. Configure Credentials:
   • FTP Node: Click on the "FTP" node and configure your FTP server credentials.
   • Embeddings OpenAI Node: Click on the "Embeddings OpenAI" node and set up your OpenAI API key credential.
   • Qdrant Vector Store Node: Click on the "Qdrant Vector Store" node and configure your Qdrant connection details (host, port, API key).
3. Customize Nodes (Optional):
   • FTP Node: Adjust the "Operation" (e.g., "List Files", "Download File") and "Path" settings to target the specific JSON files on your FTP server.
   • Character Text Splitter Node: Modify chunk size and overlap parameters as needed for your specific data and embedding requirements.
   • Qdrant Vector Store Node: Configure the "Collection Name" and any other Qdrant-specific settings.
4. Execute the workflow:
   • Click the "Execute Workflow" button in the "When clicking ‘Execute workflow’" (Manual Trigger) node to run the workflow manually.
   • Alternatively, you can activate the workflow and set up a schedule or webhook trigger if you need it to run automatically.