Abstract: This presentation focuses on the advanced utilization of Retrieval-Augmented Generation (RAG) and multi-agent Large Language Models (LLMs) for the synthesis of scientific knowledge. Using the development of WikiCrow as a case study, we explore how these technologies can efficiently curate and synthesize vast amounts of scientific literature, a task traditionally bottlenecked by information retrieval and summarization of 100s of millions of source documents.
We will discuss the architecture and mechanics of multi-agent LLM systems like PaperQA, which underpin WikiCrow. This includes their ability to perform complex tasks such as identifying relevant scientific papers, parsing and summarizing text, and synthesizing this information into concise, accurate summaries. The focus will be on the technical strategies employed to reduce common issues like hallucinations in LLM outputs and the methods used to improve citation accuracy and relevance.
The session will also cover the challenges and strategies in evaluating the performance of such systems, highlighting the importance of information retrieval, and the hurdles in assessing the veracity of AI-generated content. We aim to provide attendees with practical insights into how RAG and multi-agent LLMs can be integrated into their own systems for more effective data processing and knowledge synthesis.
Attendees will leave with a deeper understanding of the potential and limitations of current AI technologies for knowledge synthesis. This talk is particularly suited for data scientists, AI researchers, and professionals interested in the application of LLMs and RAG systems for improving retrieval and knowledge management for AI agents and humans in scientific and research-oriented domains.
Bio: Matt Rubashkin is the Head of Engineering at Future House, a non-profit moonshot focused on building an AI Scientist to accelerate the pace of discovery across the world. Previously, Matt built out Engineering orgs at digital health companies (Ready and Cerebral), founded and led a medical speech recognition startup (LexiconAI), and was a practicing software engineer and data scientist (SVDS, Insight Data) after the completion of his graduate research at UCSF and UC Berkeley.