Video Summary
Prompting remains a foundational engineering skill for working with LLMs and agentic systems.
Start debugging with evaluations: include a control case, edge cases, and capability/hand-off checks.
Clean up prompts (role, policies, remove patches) before targeted fixes — apply prompt hygiene.
Define an explicit output contract or structured schema for consistent, parseable outputs.
Avoid overly cautious instructions that cause output deflection; be specific about authoritative sources and exceptions for legacy plans or data gaps.
Include a control case that should always pass, edge cases that capture prior failures, and tests that verify the model understands its capability boundaries and when to hand off to humans.
An output contract (or structured schema) enforces consistent, machine-parseable outputs and prevents format drift—critical when responses must adhere to nested formats like JSON or XML.
Overly cautious instructions such as 'never give wrong information' can push the model to avoid answering and redirect users; revising guidance to cite authoritative customer data can reduce unnecessary deflection.
Instructions alone don't add capability—explicitly direct the agent to call a defined tool (via the API/harness) for calculations so the agent can reliably perform tasks like proration.
Escalate when billing errors risk refunds or customer trust; prompts should balance cost concerns with the necessity of human intervention rather than discouraging escalation outright.
“Prompting is arguably one of the first skills, if not the first skill, that we had to learn as engineers when we first started to work with LLMs.”
Prompting remains a foundational skill for engineers when developing AI systems. It is crucial for ensuring effective interaction with large language models (LLMs).
Understanding how to create and refine prompts can significantly impact the success of AI implementations, making it one of the most critical skills today.
“Today we're going to discuss some best practices in the context of two practical scenarios that you're probably encountering at work.”
The discussion will cover two main scenarios: maintaining an existing prompt that is no longer performing well, and building a new prompt from scratch for an agentic use case.
These scenarios reflect common challenges that engineers face in their day-to-day work, emphasizing the need for continuous learning and adaptation in prompt engineering.
“In order to start unpacking that question, we need a starting point, and that starting point is evaluations.”
Evaluations are essential for understanding whether changes to prompts lead to improved performance. They provide a rigorous testing framework to assess effectiveness.
Different models have varying capabilities, and screen outputs against a defined evaluation suite can help pinpoint issues and ascertain where to apply improvements.
“The three key cases include having a control case, edge cases, and understanding of the model's capabilities.”
The evaluation should consist of three important types of test cases: a control case that always passes, edge cases that highlight potential failures, and ensuring the model understands its own limitations.
This structured approach helps prevent regression in performance and ensures comprehensive coverage of prompt behaviors under varying conditions.
“So, in the example that we're going to be looking at today, we'll be using a prompt for a customer support bot for a Telco company called Meridian Mobile.”
The selected example is a customer support bot for a telecommunications company, which includes specific test cases aimed at evaluating performance around issues commonly faced in customer support interactions.
Test cases will encompass routine inquiries, edge scenarios, and the model's ability to deal with potential errors effectively.
“So, simply clearing up the prompt with a better structure, with a better role description, has improved the performance.”
An effective prompt should be well-structured and clear, which includes defining roles, separating guidelines from policies, and removing any unnecessary information.
Cleaning up prompts is a best practice that can enhance performance and should be revisited throughout the lifecycle of prompt development. It cultivates better comprehension in the model, leading to more accurate outputs.
"Creating an output contract is a key best practice to follow if you're struggling with output format consistency."
Establishing an output contract is crucial, especially if you encounter challenges with maintaining a consistent output format.
The example discussed involves a customer support bot, which is expected to respond in a conversational tone; however, this becomes critical when dealing with complex output formats like nested JSON documents.
To enhance the prompt effectively, new sections can be added to define required output formats. In this case, specifying XML tags for response output is beneficial.
It's important to recognize that adjustments can also be made within the system harness to improve consistency, such as implementing a stop sequence in API calls to indicate when the model should cease generating responses.
"For more complex output schemas, using structured outputs can ensure consistency in a more programmatic way."
After refining the initial output, two test cases began to pass consistently, while three specific failure modes emerged: proration, billing errors, and hotspot issues.
Focusing on the hotspot question, the model’s goal is to accurately convey the amount of hotspot data available to a customer on a legacy plan.
"The instruction 'Never give a customer the wrong information' is what the bot has been optimizing for."
The model's tendency to redirect customers to a URL for information rather than providing it directly stemmed from an overly cautious instruction designed to prevent misinformation in earlier versions of the model.
As models evolve and improve at instruction following, such restrictive phrases can lead to unnecessary deflection rather than delivering accurate responses.
The revised instruction encourages acknowledging that customers on grandfathered plans have different allowances, reinforcing that the provided customer information is the source of truth.
"Instructions do not add capability; they only direct behavior."
When a customer inquires about their bill after upgrading, the model struggles to manage the calculation effectively. The existing prompts guide it to avoid vague answers, but do not enhance its computational ability.
To rectify this, the prompt should explicitly instruct the model to utilize a specific tool for calculating proration when necessary. This not only provides capability but also enhances the reliability of responses.
Implementing this requires defining the tool in the API and ensuring the tool's calculations are accurately deployed.
"If you escalate this incorrectly, it could cost you a refund as well as customer trust."
In scenarios involving billing conflicts, the model should prioritize escalating the issue to a human agent, rather than simply attempting to diagnose the problem itself.
The initial prompt's guidelines leaned too heavily towards avoiding escalations by highlighting the cost associated with these actions, which could lead to neglected necessary escalations.
A balanced instruction is vital, weighing the cost of not escalating against the potential loss of customer trust and revenue, thus equipping the model with a more nuanced approach to decision-making.
"General hygiene principles can provide an initial uplift to the prompt by removing redundant instructions."
It is essential to apply general hygiene principles when creating prompts, as they enhance prompt effectiveness by eliminating unnecessary instructions. These redundant instructions may have been developed as quick fixes for previous models' behaviors.
This process helps in providing a model with reliable tools for executing specific tasks, ensuring optimal functionality.
"When building a new agent from scratch, we need to consider not just the prompt, but also the model and the harness we're using."
Developing a new agent requires a holistic approach that encompasses the prompt, the choice of model, and the harness that connects them.
The example of creating a retail staff schedule will guide the process and demonstrate the necessary considerations when starting from ground zero.
"We will compare several approaches to explore the impact of the different areas involved in prompt development."
A series of tests will be conducted using various methods to assess how different approaches influence the scheduling task at hand.
A web app has been set up to visualize the scheduling problem, which includes eight employees and constraints that must be satisfied across all scenarios.
"In our tests, we initially ran a simple model, but all cases failed in the first iteration."
The initial evaluation of a basic model (Sonnet 4.6) showed that it failed to generate satisfactory schedules, producing numerous violations.
By switching to a more advanced reasoning model (Opus 4.7), the number of violations decreased, indicating an improvement, although the tests still resulted in failures.
"We're looking to see if we can optimize the cost-latency trade-off further."
The exploration of optimizing the balance between token usage and latency is crucial for enhancing performance.
Although increasing the reasoning capability of a model resulted in an improved output, it also led to greater resource consumption.
"We will implement a generate-evaluate-repair loop to streamline violations checking."
A new strategy involves using a three-step process where the generator drafts a schedule first, followed by a separate prompt to identify specific violations, and a final repair prompt that attempts to rectify these violations.
This approach allows for clearer workflow management and more effective problem-solving by isolating tasks into manageable components.
"Isolating different tasks makes it easier to achieve repeatable and reliable results."
The separation of tasks into three distinct prompts has shown effectiveness in addressing the scheduling problem with higher accuracy and efficiency.
Rather than relying on one complex prompt, breaking down tasks facilitates better management and reliability in outputs, ultimately leading to optimized performance in meeting requirements.
