Evaluate AI Agent Response Relevance using OpenAI and Cosine Similarity

This n8n workflow automates the process of evaluating the relevance of AI agent responses using OpenAI's embedding models and cosine similarity. It's designed to help assess the quality of AI-generated content against a predefined "golden answer" or expected response.

What it does

This workflow performs the following key steps:

1. Triggers Evaluation: Initiates the workflow for each row in a dataset, typically containing a user query, an AI agent's response, and a "golden answer" for comparison.
2. Sets up Evaluation Fields: Prepares the data, extracting the AI agent's response and the golden answer for subsequent processing.
3. Generates OpenAI Embeddings:
   • Sends the AI agent's response to OpenAI to generate a vector embedding.
   • Sends the golden answer to OpenAI to generate a vector embedding.
4. Calculates Cosine Similarity: Uses custom JavaScript code to compute the cosine similarity between the two generated embeddings. This metric indicates how semantically similar the AI agent's response is to the golden answer.
5. Records Evaluation Metrics: Stores the calculated cosine similarity score as an evaluation metric, allowing for quantitative analysis of response relevance.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running n8n instance.
• OpenAI API Key: An API key for OpenAI with access to embedding models (e.g., text-embedding-ada-002). This key needs to be configured as an n8n credential.
• Evaluation Dataset: A dataset (e.g., CSV, Google Sheet, database) containing at least:
  • ai_agent_response: The response generated by the AI agent.
  • golden_answer: The expected or ideal response for the given query.

Setup/Usage

1. Import the Workflow:
   • Download the provided JSON file for this workflow.
   • In your n8n instance, click "Workflows" in the left sidebar.
   • Click "New" -> "Import from JSON" and upload the downloaded JSON file.
2. Configure Credentials:
   • Locate the "OpenAI Chat Model" nodes (1153) within the workflow.
   • Ensure they are configured with your OpenAI API Key credential. If not, create a new OpenAI API credential and select it for these nodes.
3. Configure the Evaluation Trigger:
   • The "When fetching a dataset row" (1300) node is an Evaluation Trigger. This node is designed to be used within n8n's evaluation framework.
   • When running an evaluation, you will select your dataset and map the relevant columns to ai_agent_response and golden_answer as expected by the "Edit Fields" (38) node.
4. Activate the Workflow:
   • Once configured, activate the workflow by toggling the "Active" switch in the top right corner of the workflow editor.
5. Run an Evaluation:
   • Navigate to the "Evaluations" section in n8n and create a new evaluation.
   • Select this workflow and your dataset.
   • Map the input fields (ai_agent_response, golden_answer) from your dataset to the corresponding fields in the "Edit Fields" node.
   • Run the evaluation to get relevance scores for your AI agent's responses.