Abstract: 

This hands-on workshop shows you how to troubleshoot issues with your LLM applications using Phoenix, an open-source Python package and application from Arize AI. In particular, participants will learn how to detect issues that arise during retrieval-augmented generation (RAG), the process of fetching documents from a knowledge base and inserting those documents into a prompt to an LLM. Retrieval-augmented generation is a powerful technique that enhances the question-answering capabilities of LLMs, but that also introduces a new point of failure; the retrieval of irrelevant documents can leave the LLM without the information needed to generate a satisfactory response and can increase the propensity of the LLM to “hallucinate” or otherwise produce undesirable output.

In this workshop, participants will:

- Download and run an LLM chatbot application built using LlamaIndex, an LLM orchestration framework, to answer questions about the Arize AI documentation,
- Visualize both query and knowledge base embedding distributions in lower dimensions,
- Automatically identify areas of user interest that are not answered by the knowledge base,
- Surface poorly performing queries based on user feedback and LLM-assisted ranking metrics.

PREREQUISITES:
Jupyter Notebook: https://bit.ly/llama-index-phoenix-tutorial

Phoenix GitHub: https://github.com/Arize-ai/phoenix

Tools: Phoenix, LlamaIndex

Bio: 

Xander Song is a Machine Learning Engineer and Developer Advocate at Arize AI and one of the creators of Phoenix, a popular notebook-first python library that leverages embeddings to uncover problematic cohorts of LLM, CV, NLP and tabular models. Before joining Arize, Song worked as a machine learning engineer at early stage AI startups. He is based in Oakland, California.