Mary Newhauser
Machine Learning Engineer at Weaviate.
Templates by Mary Newhauser
Build a weekly AI trend alerter with arXiv and Weaviate
Build a Weekly AI Trend Alerter with arXiv and Weaviate Ditch the endless scroll for AI trends. Meet Archi, your personal AI research assistant that hits you up once a week with everyone you need to know. 🧑🏽🔬 This workflow scrapes AI and machine learning article abstracts from arXiv, enriches them with topic categories using a LLM, and embeds them in a Weaviate vector store. The vector store is then used as a tool for agentic RAG to write a concise, easy-to-read summary of the week in AI research. The final output is a short, weekly email sent to the address of your choice that summarizes key AI research trends and future research directions, with links directly to the most interesting and impactful arXiv papers of the week. Who it's for This workflow is for anyone who can't keep up with all the latest AI advances. Coding skills are not required. How it works This is a contiguous workflow that can be summarized in two main parts: a data pipeline that fetches and embeds articles in Weaviate, and an agentic workflow that generates a weekly email summary. Part 1: Automatically fetch newly published articles on a weekly basis Fetch article abstracts (and metadata) from arXiv's free API Pre-process abstract data Enrich each article with a primary topic, secondary topics, and estimated potential impact of the research using a LLM Post-process data Insert data and embeddings into Weaviate Part 2: Use an AI Agent and Weaviate to generate a weekly summary email Add Weaviate as a Tool to an AI agent node Query Weaviate, agentically, to generate a report on the most important research trends of the week Post-process data Send the summary via email Prerequisites An existing Weaviate cluster. You can view instructions for setting up a local cluster with Docker here or a Weaviate Cloud cluster here. API keys to generate embeddings and power chat models. We use a combination of OpenRouter and OpenAI models. Feel free to switch out the models as you like. An email address with STMP privileges. This is the address the email will come from. In this demo we use a personal Gmail address. You can create a new credential to link a STMP Account using these instructions. Self-hosted n8n instance. See this video for how to get set up in just three minutes. How to run the workflow Go through the prerequisites, creating a Weaviate cluster (can be local or cloud), downloading self-hosted n8n, creating STMP privileges for your email account, and adding your API keys and other credentials. Select the embedding and chat models you'd like to use. Enter the email addresses you want to send the email from and to. Let it rip. Workflow output The output for this workflow is a weekly email that summarizes key research trends and future research directions based on AI and ML papers published on arXiv. Here's an example of a summary email: Hey there, Here's a quick rundown of the key trends in Machine Learning research from the past week. Key Research Trends This Week This week saw significant advancements in retrieval-augmented systems, foundation models for specialized domains, and techniques balancing efficiency with performance. Advanced RAG Architectures: Researchers are developing sophisticated RAG frameworks that go beyond simple document retrieval, with AdaPCR introducing passage combination retrieval and UrbanMind proposing a framework for urban intelligence with multilevel optimization. Foundation Models for Tabular Data: The Real-TabPFN shows that targeted continued pre-training on real-world datasets can significantly boost the performance of foundation models for tabular data, outperforming models trained on broader, potentially noisier datasets. Efficiency-Focused Techniques: Researchers are developing resourceful methods that maintain performance without expensive computations, like logit reweighting for topic-focused summarization and strategic querying for privacy-preserving personalization. Future Research Directions Based on current trends, we expect to see the following developments in the near future: Explainable RAG Systems: Following the source attribution work in RAG systems, we can expect more research into making complex retrieval systems transparent and explainable for users. Cross-Domain and Cross-Modal Fusion: The promising performance of vision-language and code-specialized LLMs in retrieval tasks points toward unified retrievers capable of handling text, code, images, and multimodal content. Data-Centric Synthetic Generation: As shown by work on synthetic relational tabular data, we'll likely see more sophisticated approaches to generating high-quality synthetic data for pre-training foundation models in specialized domains. This week highlights how researchers are making AI more efficient, explainable, and applicable to specialized domains. Look out for more developments in RAG systems, tabular foundation models, and privacy-preserving AI techniques in the coming weeks. Until next week, Archi Want to make it better? Feel free to tweak, build on, or completely reconfigure this workflow. If you come up with something cool, let us know and we might just share it with our community! 💚
By Mary NewhauserDocument Q&A with RAG: Query PDF content using Weaviate and OpenAI
RAG over a PDF with Weaviate This workflow allows you to upload a PDF file and ask questions about it using the Question and Answer Chain and the Weaviate Vector Store nodes. Who it's for This workflow is the simplest possible implementation of RAG with Weaviate in n8n. It's intended to act as an extendable template for RAG over your own documents. Prerequisites An existing Weaviate cluster. You can view instructions for setting up a local cluster with Docker here or a Weaviate Cloud cluster here. API keys to generate embeddings and power chat models. We use OpenAI, but feel free to switch out the models as you like. Self-hosted n8n instance. See this video for how to get set up in just three minutes. How it works Part 1: Manually upload data In this example, we manually upload a 100+ page article from arXiv called "A Survey of Large Language Models". But you can replace this with your own more advanced data pipeline, if you wish. Part 2: Embed and load data into Weaviate collection Here, we generate embeddings for the full-text of the article and store them in Weaviate. Part 3: Perform RAG over PDF file with Weaviate In this part of the workflow, you can enter your query by running the Chat Node and get a RAG response grounded in context via the Question and Answer Chain node. How to run the workflow Go through the prerequisites, creating a Weaviate cluster (can be local or cloud), downloading self-hosted n8n, and adding your API keys and other credentials. Select the embedding and chat models you'd like to use. Upload a PDF file you want to ask questions about. Execute the rest of the workflow.
By Mary Newhauser