n8n Workflow: Run Apache Airflow DAG and Retrieve XCom Value

This n8n workflow demonstrates how to trigger an Apache Airflow DAG and then retrieve a specific XCom value associated with the DAG run. This is useful for orchestrating complex data pipelines or backend processes from n8n, and then using the results within subsequent n8n steps.

What it does

This workflow performs the following steps:

1. Receives Trigger: It starts when executed by another workflow, acting as a sub-workflow or a callable component.
2. Triggers Airflow DAG: It makes an HTTP request to an Apache Airflow API endpoint to trigger a specific DAG.
3. Waits for DAG Completion: It then waits for a configured amount of time (e.g., to allow the DAG to complete).
4. Checks DAG Status: After waiting, it makes another HTTP request to the Airflow API to check the status of the triggered DAG run.
5. Conditional Logic (If): It uses an "If" node to check if the DAG run was successful.
6. Retrieves XCom Value (if successful):
   • If the DAG run was successful, it makes an HTTP request to retrieve a specific XCom value from the completed DAG run.
   • It then processes this XCom value using a "Code" node to extract and format the desired data.
   • Finally, it uses an "Edit Fields (Set)" node to set the extracted XCom value as a field in the workflow's output.
7. Handles Failed DAG Runs (if unsuccessful):
   • If the DAG run was not successful, it stops the workflow and throws an error, indicating the failure.

Prerequisites/Requirements

• Apache Airflow Instance: A running Apache Airflow instance with a DAG configured that can be triggered via its REST API and produces an XCom value.
• Airflow API Access: Appropriate API endpoints and authentication details (e.g., API key, basic auth) for your Airflow instance.
• n8n HTTP Request Node: This workflow heavily relies on the HTTP Request node for interacting with the Airflow API.
• Knowledge of Airflow XComs: Understanding how XComs work in Airflow is beneficial for configuring the retrieval step.

Setup/Usage

1. Import the workflow: Import this JSON definition into your n8n instance.
2. Configure Airflow API Endpoints:
   • Update the "HTTP Request" nodes (for triggering, checking status, and retrieving XCom) with your specific Apache Airflow API URLs.
   • Ensure proper authentication is configured for these HTTP requests (e.g., using n8n credentials for Airflow API tokens or basic authentication).
3. Adjust DAG ID and XCom Key:
   • In the "HTTP Request" node that triggers the DAG, specify the dag_id you want to run.
   • In the "HTTP Request" node that retrieves the XCom, specify the correct dag_id, task_id, and key for the XCom value you wish to retrieve.
4. Configure Wait Time: Adjust the "Wait" node's duration to a suitable time that allows your Airflow DAG to complete its execution.
5. Customize Code Node: If the structure of your XCom value is different, you might need to adjust the JavaScript code in the "Code" node to correctly parse and extract the desired data.
6. Integrate with a Parent Workflow: Since this workflow starts with "When Executed by Another Workflow," you will need a separate parent workflow that uses the "Execute Workflow" node to call this workflow. The parent workflow can pass parameters to this workflow and receive the XCom value as output.