Compare GPT-4, Claude & Gemini Responses with Contextual AI's LMUnit Evaluation

This n8n workflow provides a robust framework for comparing the responses of different Large Language Models (LLMs) like GPT-4, Claude, and Gemini. It leverages n8n's Langchain nodes to interact with these models and a custom Code node to evaluate their responses contextually using LMUnit. This allows for a systematic and automated way to assess the quality and relevance of AI-generated content based on a given prompt and context.

What it does

This workflow automates the following steps:

1. Receives Chat Message: Initiates the workflow upon receiving a chat message, which serves as the user's prompt.
2. Prepares Context (Edit Fields): Sets up the initial context and prompt for the LLMs. This can include system instructions or specific data points.
3. Loops Over LLM Models: Iterates through a predefined list of LLMs (OpenAI, Google Gemini, Anthropic) to get responses from each.
4. Generates OpenAI (GPT-4) Response: Sends the prompt and context to OpenAI's GPT-4 model and captures its response.
5. Generates Google Gemini Response: Sends the prompt and context to Google Gemini and captures its response.
6. Generates Anthropic (Claude) Response: Sends the prompt and context to Anthropic's Claude model and captures its response.
7. Merges Responses: Combines the responses from all LLMs into a single data structure for unified processing.
8. Evaluates Responses with LMUnit (Code): Executes custom JavaScript code to evaluate each LLM's response against the original prompt and context using LMUnit. This step assigns a score or rating based on predefined criteria.
9. Waits (Optional Delay): Introduces a configurable delay, potentially to manage API rate limits or allow for external processing.
10. Responds to Chat: Sends the evaluation results and the LLM responses back to the user via the chat interface.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running instance of n8n.
• OpenAI API Key: For interacting with OpenAI's GPT-4 model.
• Google Gemini API Key: For interacting with Google Gemini.
• Anthropic API Key: For interacting with Anthropic's Claude model.
• LMUnit Integration: The Code node uses LMUnit for evaluation. Ensure LMUnit is correctly set up or that the custom code is adapted to your preferred evaluation method.

Setup/Usage

1. Import the workflow: Download the JSON provided and import it into your n8n instance.
2. Configure Credentials:
   • For the "OpenAI" node, configure your OpenAI API credentials.
   • For the "Google Gemini" node, configure your Google Gemini API credentials.
   • For the "Anthropic" node, configure your Anthropic API credentials.
3. Customize the "Edit Fields" node: Adjust the initial prompt and context as needed for your specific evaluation scenarios.
4. Review the "Code" node: The LMUnit evaluation logic is within this node. You may need to modify the JavaScript code to define your specific evaluation criteria, metrics, and how LMUnit is called.
5. Activate the workflow: Once configured, activate the workflow. It will now listen for incoming chat messages.
6. Send a chat message: Interact with the "Chat Trigger" by sending a message to initiate the comparison and evaluation process. The results will be returned via the "Chat" response node.