n8n RAG Document Relevance Evaluation Workflow

This n8n workflow is designed to evaluate the relevance of documents retrieved by a Retrieval-Augmented Generation (RAG) system. It leverages Langchain nodes to process documents, generate embeddings, and interact with an AI agent, while using n8n's evaluation framework to measure performance.

What it does

This workflow automates the following steps:

1. Triggers Evaluation: Initiates the workflow for each row in a specified dataset, typically containing queries and expected relevant documents.
2. Loads Documents: Uses a "Default Data Loader" to ingest documents, likely from an external source or provided within the dataset.
3. Splits Text: Employs a "Recursive Character Text Splitter" to break down large documents into smaller, manageable chunks suitable for embedding and retrieval.
4. Generates Embeddings: Creates vector embeddings for the processed document chunks using "Embeddings OpenAI", enabling semantic search capabilities.
5. Stores Vectors: Utilizes a "Simple Vector Store" (in-memory) to store the document embeddings, making them searchable.
6. Executes AI Agent: An "AI Agent" node (likely configured with RAG capabilities) processes the input query and retrieves relevant documents from the vector store.
7. Removes Duplicates: Ensures that the retrieved documents are unique using a "Remove Duplicates" node.
8. Edits Fields: A "Set" node, named "Edit Fields", is present, suggesting data transformation or preparation before the final evaluation.
9. Performs Evaluation: The "Evaluation" node compares the retrieved documents against a ground truth (expected relevant documents) from the dataset row, calculating metrics like precision, recall, or F1-score for document relevance.
10. (Optional) OpenAI Chat Model: An "OpenAI Chat Model" node and a generic "OpenAI" node are present, which could be used by the AI Agent for generating responses or further processing, though their direct connection to the main flow is not explicitly defined in the provided JSON.
11. (Optional) No Operation: A "No Operation, do nothing" node is included, which might serve as a placeholder or for debugging purposes.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running n8n instance (self-hosted or cloud).
• OpenAI API Key: Credentials for OpenAI to use the "Embeddings OpenAI", "OpenAI Chat Model", and "OpenAI" nodes.
• Google Sheets (Optional): A Google Sheets account and credentials if the "Google Sheets" node is intended for data input or output (though it's currently disconnected).
• Langchain Nodes: Ensure the @n8n/n8n-nodes-langchain package is installed in your n8n instance.

Setup/Usage

1. Import the workflow: Download the provided JSON and import it into your n8n instance.
2. Configure Credentials:
   • Set up your OpenAI API Key credentials in n8n for the "Embeddings OpenAI", "OpenAI Chat Model", and "OpenAI" nodes.
   • If using the "Google Sheets" node, configure your Google Sheets credentials.
3. Configure Evaluation Trigger:
   • The "Evaluation Trigger" node (When fetching a dataset row) needs to be configured with the specific dataset you want to use for evaluation. This dataset should contain your test cases, including queries and ground truth relevant documents.
4. Configure AI Agent:
   • The "AI Agent" node will need to be configured with the specific RAG logic, tools, and prompts relevant to your use case. It will likely utilize the "Simple Vector Store" and "OpenAI Chat Model" nodes.
5. Configure Evaluation Node:
   • The "Evaluation" node needs to be configured to define how document relevance is measured. This includes specifying which fields from the input data represent the query, the retrieved documents, and the ground truth relevant documents.
6. Activate the workflow: Once configured, activate the workflow. It will automatically run for each row in your evaluation dataset when triggered.
7. Review Results: The "Evaluation" node will output the calculated metrics, which can then be further processed or stored (e.g., in Google Sheets) for analysis.