Troubleshooting Retrieval-Augmented Generation


This hands-on workshop shows you how to troubleshoot issues with your LLM applications using Phoenix, an open-source Python package and application from Arize AI. In particular, participants will learn how to detect issues that arise during retrieval-augmented generation (RAG), the process of fetching documents from a knowledge base and inserting those documents into a prompt to an LLM. Retrieval-augmented generation is a powerful technique that enhances the question-answering capabilities of LLMs, but that also introduces a new point of failure; the retrieval of irrelevant documents can leave the LLM without the information needed to generate a satisfactory response and can increase the propensity of the LLM to “hallucinate” or otherwise produce undesirable output.

In this workshop, participants will:

- Download and run an LLM chatbot application built using LlamaIndex, an LLM orchestration framework, to answer questions about the Arize AI documentation,
- Visualize both query and knowledge base embedding distributions in lower dimensions,
- Automatically identify areas of user interest that are not answered by the knowledge base,
- Surface poorly performing queries based on user feedback and LLM-assisted ranking metrics.

Jupyter Notebook:

Phoenix GitHub:

Tools: Phoenix, LlamaIndex


Xander Song is a Machine Learning Engineer and Developer Advocate at Arize AI and one of the creators of Phoenix, a popular notebook-first python library that leverages embeddings to uncover problematic cohorts of LLM, CV, NLP and tabular models. Before joining Arize, Song worked as a machine learning engineer at early stage AI startups. He is based in Oakland, California.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google