n8n Workflow: Automate Research Paper Collection with Bright Data

This n8n workflow automates the process of collecting research paper data from a specified URL, extracting key information, and storing it in a Google Sheet. It's designed to streamline the data acquisition for research purposes, leveraging web scraping capabilities.

What it does

This workflow performs the following steps:

1. Manual Trigger: Initiates the workflow manually when "Execute workflow" is clicked.
2. HTTP Request: Fetches the content of a specified URL (likely a research paper listing or search results page).
3. HTML Parsing: Processes the HTML content received from the HTTP Request node to extract structured data. This node is configured to specifically target elements within the HTML, such as titles, links, and other relevant information about research papers.
4. Code (JavaScript): Executes custom JavaScript code to further process or transform the extracted data. This could involve cleaning data, reformatting it, or performing additional logic before storage.
5. Edit Fields (Set): Modifies, adds, or removes fields from the incoming data. This is used to standardize the data structure before writing to Google Sheets.
6. Google Sheets: Appends the processed research paper data as new rows to a designated Google Sheet.

Prerequisites/Requirements

To use this workflow, you will need:

• n8n Instance: A running n8n instance (cloud or self-hosted).
• Google Account: A Google account with access to Google Sheets.
• Google Sheets Credential: An n8n credential configured for Google Sheets (OAuth 2.0 or Service Account).
• Target URL: The URL of the website containing the research papers you wish to scrape.
• Bright Data (Optional, Implied by Directory Name): While not explicitly present in the JSON, the directory name "5221-automate-research-paper-collection-with-bright-data--n8n" suggests that Bright Data might be used as a proxy or web scraping service in conjunction with the HTTP Request node for more robust scraping. If you intend to use Bright Data, you would need an account and potentially configure the HTTP Request node to route through their proxy network.

Setup/Usage

1. Import the Workflow: Download the provided JSON and import it into your n8n instance.
2. Configure Credentials:
   • Set up your Google Sheets credential in n8n if you haven't already.
3. Configure Nodes:
   • HTTP Request: Update the URL parameter to the specific research paper page you want to scrape. You may also need to configure headers or authentication if the target website requires it. If using Bright Data, configure proxy settings here.
   • HTML: Review and adjust the CSS selectors within this node to accurately extract the desired data (e.g., paper titles, author names, publication dates, links) from the target website's HTML structure.
   • Code: If you need specific data transformations, modify the JavaScript code within this node.
   • Edit Fields (Set): Ensure the fields are set correctly to match the columns in your Google Sheet.
   • Google Sheets:
     • Select your Google Sheets credential.
     • Specify the Spreadsheet ID of your target Google Sheet.
     • Specify the Sheet Name where the data should be appended.
     • Ensure the Operation is set to "Append Row" or a similar action.
4. Create Google Sheet: Create a new Google Sheet with appropriate column headers (e.g., "Title", "URL", "Authors", "Date") that match the data fields being output by the "Edit Fields (Set)" node.
5. Execute Workflow: Click "Execute Workflow" on the "Manual Trigger" node to run the workflow and start collecting data.