PDF Proposal Knowledge Base with S3, OpenAI GPT-4o, and Qdrant RAG Agent

This n8n workflow creates a robust knowledge base from PDF documents stored in an AWS S3 bucket. It leverages OpenAI for embeddings and a Qdrant vector store for efficient retrieval-augmented generation (RAG). The system can then be queried via a chat interface, allowing users to ask questions and receive answers based on the content of the uploaded PDFs.

What it does

This workflow automates the process of ingesting PDF documents into a RAG-powered knowledge base and enabling interactive querying:

1. Manual Trigger: The workflow is initiated manually, typically to start the ingestion process or to prepare for a chat session.
2. AWS S3 File Listing: It connects to an AWS S3 bucket and lists all PDF files within a specified directory.
3. Loop Over PDF Files: Each listed PDF file is processed individually in a batch.
4. Extract Content from PDF: For each PDF, the workflow extracts its textual content.
5. Load Document Data: The extracted text is then loaded as a document for further processing.
6. Split Text into Chunks: The document text is split into smaller, manageable chunks using a Recursive Character Text Splitter. This is crucial for efficient embedding and retrieval.
7. Generate Embeddings: OpenAI's embedding model (e.g., text-embedding-ada-002) is used to convert each text chunk into a numerical vector (embedding).
8. Store in Qdrant Vector Store: These embeddings, along with their corresponding text chunks, are stored in a Qdrant vector database, creating the searchable knowledge base.
9. Chat Trigger (Optional/Separate Flow): Once the knowledge base is built, a separate "Chat Trigger" can be used to initiate a conversational interaction.
10. AI Agent for RAG: An AI Agent (configured with an OpenAI Chat Model like GPT-4o and connected to the Qdrant Vector Store) processes user queries. It retrieves relevant information from Qdrant based on the query's embedding and then uses the OpenAI Chat Model to generate a coherent and informed response.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running n8n instance.
• AWS S3 Account: An AWS account with an S3 bucket containing your PDF documents. You'll need AWS credentials (Access Key ID and Secret Access Key) configured in n8n.
• OpenAI API Key: An OpenAI API key with access to embedding models (e.g., text-embedding-ada-002) and chat models (e.g., gpt-4o). This needs to be configured as an OpenAI credential in n8n.
• Qdrant Instance: A running Qdrant instance (either self-hosted or cloud-based). You'll need the Qdrant host URL and API key (if applicable) configured as a Qdrant credential in n8n.
• Langchain Nodes: Ensure the @n8n/n8n-nodes-langchain package is installed in your n8n instance.

Setup/Usage

1. Import the Workflow: Download the provided JSON and import it into your n8n instance.
2. Configure Credentials:
   • AWS S3: Create or update an AWS credential in n8n with your Access Key ID and Secret Access Key.
   • OpenAI: Create or update an OpenAI credential in n8n with your API Key.
   • Qdrant: Create or update a Qdrant credential in n8n with your host URL and API key.
3. Configure AWS S3 Node (ID: 307):
   • Specify your S3 Bucket Name.
   • Set the Operation to List and Resource to File.
   • Provide the Folder Path where your PDF documents are located.
4. Configure Qdrant Vector Store Node (ID: 1248):
   • Specify your Qdrant Collection Name. This is where your document embeddings will be stored.
5. Configure OpenAI Chat Model Node (ID: 1153):
   • Select your desired OpenAI Model (e.g., gpt-4o).
6. Configure AI Agent Node (ID: 1119):
   • Ensure the Language Model is set to the "OpenAI Chat Model" node and the Vector Store is set to the "Qdrant Vector Store" node.
7. Run the Ingestion Flow:
   • Activate the workflow.
   • Click "Execute Workflow" on the "Manual Trigger" node (ID: 838) to start the PDF ingestion process. This will read your PDFs, create embeddings, and store them in Qdrant.
8. Interact via Chat (after ingestion):
   • Once ingestion is complete, you can use the "Chat Trigger" node (ID: 1247) to send messages to the AI Agent. The agent will then use the Qdrant knowledge base to answer your questions.