Prompt Optimization with GPT-4 and Langchain


Most people who call themselves prompt engineers are really just blind prompting: chatting with ChatGPT and manually reviewing the results through crude trial and error. If you're planning to use a prompt at scale – as part of a template, workflow, or product – it's essential you run that prompt 20-30 times via the GPT-4 API to see how often it fails. You also need to be rigorously A/B testing your prompt against alternative approaches to find what really makes a difference to results. Langchain is a development framework for Large Language Models (LLMs) like GPT-4, and it can help you build a consistent system for running, monitoring, and measuring the performance of your prompts, so you can optimize them against your success metric.

Session Outline:

1. Metrics – Establish how you'll measure the quality of the responses from GPT-4. At the end of this lesson you'll be aware of the benefits and limitations of the various evaluation methods available in Langchain.
2. Hypothesis – Design two or more prompts that may work, based on the latest research. You'll learn how to take AI theory and apply it practically to come up with test ideas.
3. Samples – Generate responses for your prompts using against multiple test cases. Establish the best practices for working with GPT-4 as well as open-source alternatives using Langchain.
4. Results – Evaluate the performance of your prompts and use them to inform the next test. See just how important it is to be stress-testing prompts at scale, to guarantee performance.

Learning objectives: We'll be using the open-source Langchain framework, in order to work with OpenAI's GPT-4 model (paid). We'll also learn how Lanchain makes it easy to substitute GPT-4 for an open-source model and rate the relative performance.

Background Knowledge:

Python experience needed, and access to an OpenAI account. Familiarity with Langchain a plus, but not a requirement.


Mike is a data-driven, technical marketer who built a 50 person marketing agency (Ladder), and 300k people have taken his online courses (LinkedIn, Udemy, Vexpower). He now works freelance on generative AI projects, and is writing a book on Prompt Engineering for O’Reilly Media.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from Youtube
Consent to display content from Vimeo
Google Maps
Consent to display content from Google