Video Summary
Evals are systematic measurements of AI product quality—data analysis + targeted tests, not just ad-hoc 'vibes'.
Start by sampling real traces, write clear error notes, then open-code and synthesize with axial codes until theoretical saturation.
Use code-based checks for cheap, binary failure modes and LLM-as-judge for complex, nuanced failure modes—iterate and align with humans.
Assign a single domain expert (a 'benevolent dictator') to keep evaluations efficient and avoid committee paralysis.
Initial setup takes a few days; ongoing maintenance can be ~30 minutes/week. Prioritize the most business-critical evals.
Evals are systematic measurements of application quality—combining data analysis of real traces with targeted tests—to surface failure modes, prioritize fixes, and track regressions. They turn subjective 'vibe checks' into repeatable, actionable metrics.
Sample real production traces, write clear human-readable notes on errors (avoid vague terms), open-code those notes, then synthesize axial codes to cluster failure modes. Iterate until theoretical saturation—often after dozens to a hundred samples.
Use code-based evals for cheap, deterministic, binary checks (format, missing fields, obvious rule violations). Use an LLM-as-judge for nuanced, semantic, or high-level failure modes—after aligning and validating the LLM against human judgments.
Compare the LLM's binary judgments against a labeled human sample, build confusion matrices, inspect disagreements (especially rare errors), and iterate prompts until alignment meets your risk tolerance.
Appoint a single knowledgeable owner (the 'benevolent dictator'), typically a product manager or domain expert, to make fast, consistent decisions on codes and labels rather than relying on committees.
"To build great AI products, you need to be really good at building evals. It's the highest ROI activity you can engage in."
Building evaluations (evals) is crucial for developing effective AI applications. The process provides a high return on investment (ROI) and is described as a fun and engaging activity.
Participants often find themselves quickly becoming addicted to the process, as they gain valuable insights and experience in building AI applications.
Notably, most products typically require the eval process only once, after which the development can build upon initial findings to improve the product.
"There's a lot of controversy and drama around evals. Some people have been burned by evals in the past, leading them to reject the concept altogether."
A prevalent misconception is the belief that AI can conduct its own evals. However, this does not yield effective results; human oversight is still vital.
The notion of a "benevolent dictator" in the eval process suggests that rather than creating a committee that can complicate the process, one should appoint a trusted individual with domain expertise to manage the evals.
This approach helps in keeping the eval process manageable and cost-effective, preventing it from becoming a burdensome task.
"Evals are a way to systematically measure and improve an AI application; at its core, it is data analytics on your LLM application."
Evals enable systematic measurement and enhancement of AI applications through data analysis. They inform developers on performance metrics and improvement areas.
They provide a structured way to transition from guesswork and subjective judgments—like "vibe checks"—to a data-driven methodology, allowing for informed adjustments to applications as they scale.
An example highlighted involves a real estate assistant where evals might assess how accurately it responds to client inquiries, indicating areas needing improvement.
"Evals is a big spectrum of ways to measure application quality; unit tests are one way of doing this."
Evals encapsulate a broad range of methods to gauge the quality of an AI application, including the use of unit tests to verify essential functionalities.
They permit evaluation of both specific and vague tasks performed by AI assistants, which is crucial in their role in open-ended tasks that AI might be assigned.
By employing a comprehensive set of metrics through evals, developers can accurately determine how well their application meets user needs and expectations.
"It's really important that we don't think of evals as just tests. It's a common trap that a lot of people fall into because they jump straight to the test without grounding themselves in data analysis."
Evals, or evaluations, should not be rushed into as mere tests; they require a foundation of data analysis to determine what should be tested.
This process is fundamentally different from traditional software engineering, where there are clearer expectations for functionality. AI applications, especially with LLMs (large language models), are more variable and unpredictable.
It's essential to analyze data to identify issues and opportunities for improvement, particularly in complex AI applications like those seen in property management.
"This is a company that I worked with called Nurture Boss, and it is an AI assistant for property managers managing apartments."
Nurture Boss showcases the complexities and capabilities of modern AI applications, assisting property managers with tasks like handling inbound leads, customer service, and appointment bookings.
The interaction capabilities include various channels such as chat, text, and voice, along with tool calls for querying information about properties and users.
The data from Nurture Boss has been anonymized for educational purposes and will serve as the foundation for evaluating AI performance through error analysis.
"The first step in conquering data like this is just to write notes."
Error analysis is critical when working with AI applications, acting as a structured way to make sense of complex and sometimes overwhelming logs of data.
Product managers play a significant role in this step since they understand user experience and can identify when the AI's responses are inadequate.
Documenting observations and notes about the AI's performance will help in refining and improving the system’s interactions with users. This practice is particularly relevant when addressing user inquiries and managing expectations effectively.
"You can sample your data and just take a look, and it's surprising how much you learn when you do this."
Sampling allows developers to gain valuable insights without having to analyze the entire dataset. This can help identify issues quickly and enhance the development process.
Developers often find that they become "addicted" to this practice, appreciating the clarity it brings to their AI applications.
Notably, this approach highlights discrepancies between expected and observed application behavior, leading to better product development.
"This is more of a technical problem rather than the AI not doing exactly what we want."
When analyzing user interactions with an AI, it's crucial to recognize different types of errors, such as technical glitches versus AI misinterpretations.
In a specific example, a text message application displays errors when messages are garbled, affecting the conversation flow.
It's critical to document the earliest identified errors to streamline troubleshooting without getting bogged down by various minor issues.
"It's essential to manually do this error analysis yourself."
While large language models (LLMs) can assist in many tasks, relying solely on them for error analysis in AI interactions can lead to oversights.
LLMs often lack the contextual understanding needed to identify genuine product issues, such as false claims about features like virtual tours.
Thus, human expertise is invaluable for accurately assessing and correcting errors within AI systems during the development phase.
"The benevolent dictator is the person who does this note-taking and ideally they're the expert."
Assigning a single knowledgeable person—often referred to as a "benevolent dictator"—can streamline the error analysis process, avoiding committee pitfalls.
This approach ensures that decisions are made efficiently and based on informed judgment rather than consensus, which can slow down the process.
It's crucial for this person to have domain expertise, as they will guide the team in identifying relevant issues and improving the AI application.
"You don't want to say 'jank' because we want to understand what happened. You just want to say, 'did not confirm call transfer with user.'"
Clear and precise note-taking is crucial in AI evaluations to capture the essence of conversations and actions taken. Instead of using informal or vague language, detailed descriptions help convey what transpired during the interaction.
The notes should provide an overview of the situation, without requiring them to be perfect, as long as they summarize the critical details effectively.
"We recommend doing at least a hundred of these. There’s not a magic number; we say a hundred because after twenty, you will find it so useful that you will continue doing it."
Engaging in a substantial amount of note analysis, such as around one hundred instances, can help alleviate intimidation and enhance learning value from the process.
After a certain threshold, like twenty cases, users are likely to see the utility in the process, motivating them to continue further explorations and thereby refining their analysis skills.
"Theoretical saturation is when you stop because you're not uncovering new types of notes or concepts that will materially change your process."
The concept of theoretical saturation in qualitative analysis indicates the point at which no new data emerges from the analysis, suggesting that continued exploration may no longer yield significant insights.
Developing a sense of when saturation is reached takes experience, and individuals may find they can stop after fewer notes than they initially anticipated, depending on the situation.
"Now you have all these notes. You can start using AI to help you."
After compiling a substantial number of notes, individuals can leverage AI to assist in organizing and analyzing this data. Basic counting and categorization techniques can help transform raw notes into actionable insights.
Using AI tools, such as uploading CSV files containing notes, can streamline the categorization process, making it easier for users to identify patterns and extract meaningful information.
"The purpose of axial codes is to get to clusters of failure modes and figure out what is the most prevalent."
Axial codes serve as categories for the various failure modes identified in the notes, helping to condense and organize the data into understandable shapes.
The categorization aids in pinpointing common problems, which enables targeted focus on addressing the most significant issues at hand.
"You can iterate and see what works for you."
It is important to recognize that there is no single definitive prompt for creating axial codes; analysis requires customization based on user needs and experiences.
Iterating on the prompts can lead to improved categorization outputs that resonate more effectively for the individual conducting the analysis, enhancing their overall productivity.
"This would be so fun. Just sit there and find all the problems I'm running into and categorize them, then try to fix them."
"We are not inventing anything; this technique has been used to analyze stochastic systems for ages."
"You can use a lot of AI tools for this. I personally like using Jupyter notebooks."
"I collect them first and take a look at these axial codes to see if they make sense."
"The AI is doing it, so your open codes have to be detailed. You can't just say 'janky.'"
"An AI can actually say 'none of the above' in the axial code, which informs me that my axial codes are not complete."
"For most products, you do this process once and then you build on it over time."
"Now we can count how many times those different things occurred."
"There's a cost-benefit trade-off to evaluation; you don't want to get carried away with it."
"You want to ground yourself in your actual errors. You don't want to skip this step."
It's crucial to begin your evaluation process by focusing on the specific errors you've encountered rather than jumping straight into creating evaluations or tests.
Many people make the mistake of rushing into test writing without understanding where they've gone wrong, which can lead to ineffective solutions.
"There are different kinds of evaluations. One is code-based, and another is using a language model as a judge."
Evaluations can be categorized primarily into two types: code-based evaluations and using a language model (LLM) as a judge.
Code-based evaluations tend to be more cost-effective and are comparable to automated unit tests, allowing for thorough checks without extensive manual work.
When using an LLM as a judge, you are generally addressing more complex issues that cannot be easily automated, requiring different approaches for evaluation.
"When we talk about code-based versus LLM-based, we are saying that I could write a piece of code to check whether a failure mode is present."
Code-based evaluations involve writing functions to detect failure modes, such as ensuring outputs are in the correct format (e.g., JSON or Markdown).
In contrast, LLMs take on a more complex role, evaluating intricate failure modes where automated checks fall short.
The evaluation using an LLM provides a streamlined, binary result (pass or fail) for specific issues.
"The goal here is just to have a suite of tests that run before you ship to production."
Having a suite of tests is essential to ensure that your AI agent performs as intended before it goes live.
LLMs can be deployed not just for unit tests but also for continuous monitoring of real production traces, giving insights into failure rates and overall application quality.
This allows teams to maintain an ongoing understanding of how their models are performing in real-world scenarios.
"You want to do one specific failure and make it binary because we want to simplify things."
When creating evaluations, it is advisable to focus on specific failure modes and provide binary judgments (i.e., yes or no) rather than ambiguous scores.
This binary approach simplifies decision-making processes and enhances clarity for reporting metrics, thereby avoiding confusion that can arise from intermediate scoring systems.
"It's okay to use an LLM to help you create it, but put yourself in the loop."
While LLMs can assist in crafting evaluation prompts, it is vital to be actively involved in the process.
Review and modify the outputs generated by the LLM to align them fully with your objectives. This human oversight ensures that the evaluations remain relevant and effective for your specific application.
“When people lose trust in your evaluations, they'll lose trust in you.”
The prompt for evaluating AI models is just the starting point. Many people make the mistake of stopping after drafting their prompt, assuming it's good enough.
It is crucial to iterate on your evaluations and not accept the initial output from the model as the final word. Inaccuracies can lead to a loss of trust in the evaluations.
Before launching the language model (LM) as an evaluator, it's essential to align its judgments with human perspectives. This can be achieved by measuring the model's outputs against specific criteria or codes, ensuring that its assessments align with a human evaluator's judgments.
“Ninety percent agreement might look good on paper but can be misleading.”
Evaluators often report agreement percentages between the LM and human judgments. However, these metrics can be deceptive, as they may not account for rare errors that occur infrequently.
An apparently high percentage of agreement could be achieved simply by the LM bypassing difficult judgments, leading to misleading results.
It is essential for product managers to dig deeper into what "agreement" entails and to fully understand how often errors actually occur.
“You need to look at each type of error.”
A detailed examination of errors is necessary, especially in cases where the human evaluator and the LM do not align.
A matrix comparing human judgments and LM outputs should be analyzed for patterns of disagreement. This helps identify which types of errors are significant and require prompt adjustments.
If the evaluation has not undergone thorough iteration, it's a warning sign that further refinement is needed to ensure accuracy and reliability in the AI evaluations.
“PRDs are a great abstraction for thinking about this, but it's not the end-all be-all.”
Despite the utility of traditional PRDs in setting expectations and defining project scopes, they should not replace the iterative evaluation process.
Engaging in this process offers insights that can lead to revisions of the initial PRD, ensuring that it reflects real-world applications and adjustments informed by data.
Additionally, as projects evolve, unexpected failure modes will arise, emphasizing the necessity of continuous evaluation and adaptation of product expectations.
“You can't ever dream up everything in the first place.”
Insights from recent research reveal that determining evaluation rubrics in advance can be challenging, as developers often realize new criteria only after reviewing multiple outputs.
This underlines the evolving nature of AI evaluations, which requires an adaptable approach from both developers and product managers.
A successful evaluation process must remain flexible, leveraging both qualitative and quantitative data to refine and enhance the efficacy of AI models.
"You have finite resources and finite time. You can't write an eval for everything, so prioritize the ones that are the most pesky areas."
The discussion emphasizes the importance of focusing on the most troublesome areas in AI development. A developer should fix issues related to prompts if they are causing failures in desired behavior.
It's beneficial to prioritize evaluations in areas that pose significant risks to business outcomes rather than attempting to create evaluations for every potential failure.
While writing an evaluation for a resolved issue is possible, it is not always necessary, provided the problem has been fixed effectively.
"Data analysis is super powerful and will drive lots of improvements very quickly to your application."
Data analysis can be a strong catalyst for enhancing product performance, starting with basic techniques such as counting, which are accessible to everyone involved in AI development.
As developers gain more skills in data analysis, they can explore sophisticated methods to sample and analyze performance data.
Building an intuition for identifying conversational flow issues can lead to deeper investigations into unseen problems, similar to traditional analytics approaches in product management.
"After you've built your LLM judge, people tend to use it everywhere they can."
Once a Large Language Model (LLM) judge is created, developers are keen to integrate it into various components of their product, including unit tests and online monitoring systems.
Utilizing an LLM judge during testing can help ensure product reliability by enabling teams to catch and address failures early in the development process.
Successful products leverage these judges for improved monitoring, enhancing their understanding of application performance, which is often kept secret as a competitive advantage.
"I think the misconception is that people have very rigid definitions of what eval is."
A divide exists in opinions on the importance of evaluations, with some people misconstruing them as strictly unit tests or solely focused on data analysis.
The controversy may stem from past negative experiences where poorly executed evaluations led individuals to distrust them, resulting in an aversion to the concept of evaluations altogether.
Misinterpretation of discussions and experiences shared on social platforms like X or Twitter can further complicate understanding, as strong opinions often overshadow nuanced explanations about the role of evaluations in the development process.
"There's no world in which they are just being like 'I made cloud code, I'm never looking at anything.'"
Continuous monitoring is an integral yet often unacknowledged aspect of utilizing coding agents in AI products, as these companies inevitably engage in error analysis and performance tracking.
Although products might publicly downplay the importance of evaluations, it is likely that systematic monitoring practices are in place to ensure product quality through feedback from real user interactions.
The relationship between developers and the tools, like coding agents, facilitates a unique form of domain expertise that can streamline evaluation processes as developers engage with the product consistently, leading to effective immediate responses to issues.
"You don't need as much data, or feedback or exploration, because you know your eval process should look different."
A streamlined evaluation process is essential in AI development, allowing for efficiency as less data and feedback are required for effective assessments.
The ability to examine the code generated by AI allows developers to quickly discern quality, distinguishing between good and bad outputs.
"If humans on the team are doing very close data analysis, error analysis, and dogfooding, they essentially become the human evals."
Active participation from team members in data and error analysis enhances the evaluation process, acting as a form of human evaluation.
The effectiveness of evaluation can depend on how motivated and dedicated team members are to engage with the product on a deeper level.
"Dogfooding is a dangerous one because a lot of people will say they're dogfooding, but are they really?"
Many people claim to be engaging in dogfooding but lack genuine, hands-on experience, which is crucial for forming accurate assessments.
The depth of engagement needed for real feedback is often underestimated, indicating a need for a more committed approach to this practice.
"An eval is a systematic measurement of quality, and you can't really do an A/B test without the eval to compare."
The discussion around evaluations often intersects with A/B testing, as both are methods to measure and understand product performance.
A/B testing should not be conducted in isolation but rather backed by insightful evaluations to ensure that changes are informed and warranted.
"Fundamentally, you need data science to improve your applications."
Evaluations in AI products should integrate data science principles to effectively utilize available data for enhancing applications.
Despite the emergence of specific terminologies like "eval," the underlying processes and requirements remain rooted in established data science practices.
"What we hope to see is some more data science thinking in this eval process."
There is a growing recognition that improving the evaluation process requires not just generic tools but a structured approach that incorporates data science principles.
Leaders in the field emphasize the need for more individuals to adopt this mindset, moving beyond basic evaluations to a deeper analysis of product performance and errors.
"The most common misconception is, 'I can just buy a tool, plug it in, and it'll do the eval for you.'"
Many assume that AI can handle the evaluation process entirely on its own, without the need for human oversight or intervention. However, this reliance on automated tools often leads to ineffective results.
There is a significant demand from users for one-size-fits-all AI solutions, yet these do not adequately address the complexities involved in evaluations.
It is essential to recognize that human insights and critical thinking remain vital in the evaluation process to ensure its success.
"People just don't know how powerful looking at the data is."
A common oversight is neglecting to consult the collected data during evaluations. Evaluators are often surprised by the rich insights data analysis can provide.
By examining individual traces and metrics, teams can uncover the underlying issues hindering their products, leading to actionable steps for improvement.
Regularly reviewing the data not only enhances understanding but also fosters better decision-making to elevate product performance.
"Don't be scared to look at your data."
It’s crucial to approach the evaluation process with a structured mindset. Evaluators should not fear making mistakes; the larger goal is to drive actionable improvements.
Utilizing large language models (LLMs) can streamline the process by helping to organize thoughts and collect initial product requirements. This will facilitate a smoother evaluation journey.
Emphasizing the use of AI should be to enhance human capabilities rather than replace them; this enables evaluators to achieve better insights without compromising their role in the process.
"You should create your own tools to make it as easy as possible."
Developing customized tools to analyze data can significantly simplify the evaluation process. Leveraging AI can streamline these tool creation efforts, making it more accessible than ever to generate useful applications.
The goal should be to reduce friction in accessing and interpreting data, allowing for a more effective evaluation process that can yield high returns on investment.
With well-designed tools in place, organizations can more easily analyze and enhance their products, which directly correlates to business success.
"It's really not that much time."
Although there is an initial commitment of three to four days for setting up effective evaluation frameworks, this investment pays off in the long run as it leads to continued improvement.
After the initial setup, maintaining and iterating on the evaluation process takes significantly less time, often just 30 minutes per week for ongoing assessments.
Organizations should focus on embracing a mindset of continuous learning, as the insights gained from data analysis can lead to substantive advancements in product development.
"I said, you know what, I hate this email. Given your background, I just delete that."
The speaker emphasizes the necessity of providing critical feedback when discussing AI-generated content, particularly in the context of an email that lacks personalization and relevance.
They express frustration over the generic nature of the email and advocate for improvement, highlighting the importance of aligning messaging with the recipient's background.
Such critical feedback is seen as enjoyable and essential for refining the product, pushing creators to think deeper about the effectiveness of their communications.
"We go through a life cycle of error analysis, then automated evaluators, then how to improve your application."
The course discussed includes various components focused on error analysis and application improvement, providing a comprehensive approach to mastering AI evaluations.
Unique topics included in the syllabus touch on building interfaces for error analysis and optimizing costs by replacing expensive models with more affordable alternatives while maintaining quality.
The course aims to equip students with practical skills and knowledge in AI evaluations which are necessary for product improvement.
"There's a 60-page book that walks through the entire process in detail of how to do evaluations."
An extensive resource accompanies the course, eliminating the need for students to take copious notes, as all relevant content has been meticulously documented in a clear, organized manner.
A notable perk of the course is an AI tool available to students, which aggregates all prior discussions and materials, making it easier to access important insights and guidance related to evaluations.
Students receive free unlimited access to this AI for ten months, thereby enhancing their learning experience by providing immediate assistance and answers based on the course content.
"We also have a Discord of all the students who have ever taken the class, and that Discord is so active."
The presence of an active community through Discord allows students to engage with one another, facilitating networking and support beyond the course material.
This community aspect ensures that students receive ongoing feedback and a platform to discuss concepts, ask questions, and share experiences, enriching their learning journey.
The high level of engagement within the Discord group demonstrates the course's value in fostering collaboration and continuous dialogue among participants.
"The UX is outstanding, and there's a lot of love that went into that. It's really impressive as a terminal application that is that nice."
"Keep learning and think like a beginner."
"Always try to think about the other side's argument."
"Shreya is one of the wisest people that I know, especially for being so young. I'm just really impressed by that all the time."
"My favorite thing about Hamal is his energy. I often think I would start caring much less about Evals if not for Hamal."
"You can find the course if you Google AI evals for engineers and product managers or just AI evals course."
"We would love other people to teach evals. Any kind of blog posts or writing that you want to share, we would be delighted to help reshare that or amplify that."
