n8n ETL Pipeline for Text Processing

This n8n workflow provides a robust ETL (Extract, Transform, Load) pipeline for processing text data, incorporating natural language analysis, conditional logic, and database storage, with optional notifications. It's designed to be triggered on a schedule, fetch data from a database, analyze it, make decisions based on the analysis, and then store or notify as needed.

What it does

This workflow automates the following steps:

1. Schedules Execution: The workflow is triggered periodically by a Cron job.
2. Extracts Data from MongoDB: It connects to a MongoDB database to retrieve text data.
3. Extracts Data from PostgreSQL: It also retrieves data from a PostgreSQL database.
4. Analyzes Text with Google Cloud Natural Language: The extracted text data is sent to Google Cloud Natural Language for sentiment analysis or other text processing.
5. Transforms Data: A Set node is used to manipulate or prepare the data, likely combining or reformatting the results from the database queries and natural language processing.
6. Applies Conditional Logic: An If node evaluates the processed data (e.g., sentiment scores) to determine the next steps.
7. Performs No Operation (Placeholder): If the condition in the If node evaluates to false, a "No Operation" node is executed, indicating that no further action is taken for that specific data item.
8. Sends Notifications (Conditional): If the condition in the If node evaluates to true, a message is sent to a Slack channel.
9. Posts to X (Formerly Twitter) (Conditional): If the condition in the If node evaluates to true, a tweet is posted to X.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running n8n instance.
• MongoDB Account/Instance: Access to a MongoDB database.
• PostgreSQL Account/Instance: Access to a PostgreSQL database.
• Google Cloud Account: With the Natural Language API enabled and configured credentials.
• Slack Account: To send notifications.
• X (Formerly Twitter) Account: To post tweets.

Setup/Usage

1. Import the Workflow: Import the provided JSON into your n8n instance.
2. Configure Credentials:
   • Set up credentials for MongoDB.
   • Set up credentials for PostgreSQL.
   • Configure Google Cloud Natural Language credentials (typically an OAuth 2.0 or service account credential).
   • Set up Slack credentials.
   • Set up X (Formerly Twitter) credentials.
3. Customize Nodes:
   • Cron Node: Adjust the schedule as per your requirements.
   • MongoDB Node: Configure the database, collection, and query to extract the desired text data.
   • PostgreSQL Node: Configure the database, table, and query to extract the desired text data.
   • Google Cloud Natural Language Node: Specify the text field to analyze and the desired analysis type (e.g., sentiment, entities).
   • Edit Fields (Set) Node: Modify the fields to transform or combine data as needed.
   • If Node: Define the conditions based on the output of the Google Cloud Natural Language analysis (e.g., if sentiment.score < -0.5).
   • Slack Node: Customize the message content and target channel.
   • X Node: Customize the tweet content.
4. Activate the Workflow: Once configured, activate the workflow to start the scheduled text processing.