Batch Process Data with Redis-Powered Debouncing System

This n8n workflow provides a robust system for batch processing data with a built-in debouncing mechanism powered by Redis. It's designed to handle incoming data, prevent redundant processing of identical items within a short timeframe, and then trigger a separate workflow for actual batch processing.

What it does

This workflow automates the following steps:

1. Receives Data: It's designed to be triggered by another workflow, indicating that new data items are ready for processing.
2. Splits Data: If multiple data items are received, it splits them into individual items for separate processing.
3. Hashes Data: Each incoming data item is hashed using the Crypto node to create a unique identifier. This hash is crucial for the debouncing mechanism.
4. Checks Redis for Debounce: It queries a Redis instance to check if an item with the same hash has been processed recently.
5. Debounces Duplicate Items:
   • If a duplicate is found: The workflow pauses for a configurable duration (e.g., 5 minutes) to allow the previous processing of that item to complete, effectively debouncing rapid, identical requests.
   • If not a duplicate: The item proceeds immediately.
6. Stores Hash in Redis: For unique items, their hash is stored in Redis with an expiration time, marking them as "in-process" for the debouncing system.
7. Triggers Batch Processing Workflow: It then triggers a separate, dedicated workflow (referred to as "Batch Processing Workflow" in the notes) to handle the actual processing of the unique data item.

Prerequisites/Requirements

• n8n Instance: A running n8n instance.
• Redis Instance: Access to a Redis server. You'll need to configure a Redis credential in n8n.
• "Batch Processing Workflow": A separate n8n workflow that this workflow will trigger to perform the actual batch processing of the data. This workflow should be designed to accept data from an Execute Workflow Trigger node.

Setup/Usage

1. Import the Workflow: Import the provided JSON into your n8n instance.
2. Configure Redis Credentials:
   • In the "Redis" node, click on "Credentials".
   • Add or select your Redis credential, providing the necessary host, port, and authentication details for your Redis server.
3. Configure Debounce Logic:
   • In the "Wait" node (connected to the "TRUE" branch of the "If" node), adjust the "Wait for" duration if you want to change how long the workflow pauses for debounced items. The default is 5 minutes.
   • In the "Redis" node that sets the key (after the "If" node), configure the EXPIRE time for the Redis key. This determines how long an item is considered "in-process" for debouncing purposes.
4. Configure Crypto Node: The "Crypto" node is set to hash the entire incoming item. If you only want to debounce based on a specific field, adjust the "Value" field in the "Crypto" node to reference that specific field (e.g., {{ $json.id }}).
5. Set up the "Batch Processing Workflow": Ensure you have another n8n workflow that is triggered by an Execute Workflow Trigger node and is ready to receive and process the data items.
6. Connect to a Trigger: This workflow is designed to be executed by another workflow. You will need to set up an Execute Workflow node in your primary workflow to trigger this "Batch Process Data with Redis-Powered Debouncing System" workflow, passing the data you wish to process.

This workflow is a powerful tool for building resilient data processing pipelines, preventing unnecessary load on downstream systems caused by rapid, identical data inputs.