domain_3
Foundation Model Design Considerations
When you design a solution around a foundation model, you have to balance many important factors. The full list includes cost, modality, quality, latency, supported languages, model size, model complexity, context length, input and output limits, customization, safety, compliance, regional availability, prompt caching, and integration.
Prompt caching is a technique that saves the stable, unchanging beginning of a prompt—like long system instructions or policy documents. When many requests share that same prefix, the model reuses it instead of reprocessing it each time.
Prompt caching is especially useful when you have a large, fixed prefix that appears in many prompts. The benefits are lower latency and lower input-processing cost.
Q What are the key design considerations when building a foundation model solution?
A You need to weigh cost, modality, quality, latency, supported languages, model size, model complexity, context length, input and output limits, customization, safety, compliance, regional availability, prompt caching, and integration.
Q What is prompt caching, and when is it most useful?
A Prompt caching reuses the stable prefix of a prompt across multiple requests. It is most useful when many requests share a large, stable prefix, such as system instructions or policy documents. It reduces both latency and cost.
Inference Parameters
Temperature controls how random or predictable a model's output is. Think of it like a creativity dial. Low temperature makes the model pick the most likely words, giving consistent, factual answers. High temperature lets it explore less likely words, which is great for brainstorming or creative writing.
Use low temperature for tasks like classification, answering factual questions, or generating structured output like JSON. Use high temperature for creative tasks like story writing or idea generation.
Top-P, also called nucleus sampling, selects from the smallest set of tokens whose combined probability reaches a threshold. For example, if you set Top-P to 0.9, the model only considers the most probable tokens that together make up 90% of the probability mass. Lower Top-P means fewer choices, more focused output. Higher Top-P means more diversity.
Top-K simply limits the model to the K highest probability tokens. If K is set to 50, only the 50 most likely next words are considered. Lower K gives narrower choices, higher K broader. Not every model supports Top-K.
Top-P and Top-K both limit token choices, but they work differently. Top-K uses a fixed number of tokens, regardless of probability distribution. Top-P adapts to the shape of the distribution, choosing a variable number of tokens until the cumulative probability hits the threshold. This means Top-P can be more dynamic, while Top-K is simpler and more predictable.
Input length refers to the context window—how much text you can feed into the model. Longer contexts allow more information, but too much irrelevant text can distract the model and reduce accuracy.
Maximum output tokens sets a cap on how long the generated response can be. It depends on the model and the input length, because the total tokens available (input plus output) is limited by the model's context window.
Stop sequences are specific strings that tell the model to stop generating. For instance, you might set a stop sequence of 'END' to prevent the model from rambling.
Repetition penalties discourage the model from repeating the same words or phrases. A higher penalty makes the model less likely to repeat itself, which helps keep output fresh and varied.
Q When would you set a low temperature for a model?
A You would use a low temperature when you need predictable, consistent, and factual outputs, such as for classification, extracting structured data, or answering questions with a single correct answer.
Q What is the effect of lowering the Top-P value?
A Lowering Top-P restricts the model's choices to only the most probable tokens, making the output more focused and less diverse.
Q How does Top-K differ from Top-P in controlling token selection?
A Top-K uses a fixed number of tokens, say the top 50, regardless of their probability distribution. Top-P, on the other hand, selects a varying number of tokens until their total probability reaches a threshold, so it adapts to the confidence of the model's predictions.
Q Why might a longer input length hurt performance?
A A longer input can include irrelevant or distracting information that confuses the model, causing it to lose focus on the important parts and produce less accurate answers.
Prompt Engineering Structure
A well-built prompt is like giving a new team member a clear briefing. It sets them up for success by telling them who to be, what to do, and how to deliver.
A strong prompt typically includes six elements: a role or persona, clear instructions, relevant context, any constraints, the desired output format, and examples when they are helpful.
On the exam, prompts are often boiled down to four key parts. You need to recognize each part by its job and the phrasing that points to it.
Instructions — The command that tells the model what to do. Look for phrasing like 'you are to', or a direct task such as 'summarize the ticket'.
Context — The background that sets up the situation. This is the 'why' behind the task, giving the model the scenario it needs to understand the request.
Input data — The specific text or data the model needs to process. This is the raw material, like the customer ticket text or a paragraph to analyze.
Output indicator — The desired format or structure for the answer. This tells the model how to package its response, such as 'return as JSON' or 'list the reason, cause, and next action'.
Context versus Input data. Context is the background story that frames the task. Input data is the actual content you want the model to work on. For example, context might be 'you are helping a frustrated customer', while the input data is the customer's exact complaint message.
A major exam trap involves the phrase 'most operationally efficient'. When you see this, think of zero-shot prompting. Zero-shot means giving the model a task with no examples, which is the fastest and lowest-overhead approach. Do not be tricked into choosing fine-tuning or few-shot prompting when efficiency is the key word.
Q A prompt contains the phrase 'Return the reason, likely cause, and next action.' Which of the four prompt parts does this phrase represent?
A It is the output indicator, because it specifies the desired format and structure of the answer.
Q You need to generate product descriptions at scale with minimal setup and operational overhead. Which prompting strategy is the most operationally efficient?
A Zero-shot prompting. It requires no examples or fine-tuning, making it the most efficient approach.
Q In a prompt, what is the difference between the context and the input data?
A Context is the background scenario that helps the model understand the situation. Input data is the specific text the model needs to process.
Generative AI Evaluation Metrics
When you build or use generative AI models, you need ways to measure how good the outputs are. Different metrics suit different tasks. Let's walk through the most important ones.
ROUGE — measures the overlap between the generated text and a reference text. It is recall-oriented, meaning it checks how much of the important content from the reference is captured in the generated output. It's the go-to metric for summarization tasks.
BLEU — measures n-gram overlap between generated and reference text. It is precision-oriented, focusing on whether the generated translation uses words and phrases that appear in the reference. It's standard for machine translation.
How ROUGE differs from BLEU: ROUGE cares about recall—did you include the key points? BLEU cares about precision—did you use the right words? So for summarization, you want high recall; for translation, you want high precision.
BERTScore — uses deep contextual embeddings to measure how semantically similar the generated text is to the reference. Instead of exact word matches, it compares the meaning behind the words. Ideal when you care about the idea, not just the wording.
Perplexity — measures how well a model predicts the next word in a sequence. Lower perplexity means the model is more confident and fluent. But it does not measure factuality, safety, or usefulness. A smooth-talking model can still be wrong.
Cross-entropy loss — the mathematical objective minimized during training. It measures how different the model's predicted word probabilities are from the actual next word. It's a training signal, not a report-time quality score you'd show a business user.
Beyond automated metrics, real-world success is measured by business and alignment metrics. Think about task completion rate, user satisfaction, cost per interaction, productivity improvement, engagement, response latency, conversion rate, accuracy or groundedness, and return on investment. These connect model outputs to actual value.
Task engineering is about defining and structuring the work so the model can complete it reliably. When evaluating a task design, ask: Is the task clear enough? Should it be split into smaller steps? Does the model have the needed context and tools? Can success be measured? A well-engineered task leads to more dependable results.
Q Which evaluation metric is recall-oriented and best suited for summarization tasks, and why?
A ROUGE, because it measures how much of the reference content the generated summary covers, focusing on recall of important information.
Q Imagine your machine translation model produces translations that are fluent but sometimes miss details from the original. Which metric would expose this problem more clearly, BLEU or ROUGE?
A ROUGE would expose it because it is recall-oriented and penalizes missing content. BLEU, being precision-oriented, might still give a high score if the chosen words are correct, overlooking omitted details.
Q Why is perplexity alone not sufficient to evaluate a customer service chatbot's safety and factual accuracy?
A Perplexity only measures next-word prediction confidence; it doesn't check if the generated text is true, safe, or helpful. A model can have low perplexity yet still produce harmful or incorrect answers.
Q For evaluating semantic similarity where you care about meaning rather than exact words, which metric should you use?
A BERTScore, because it uses contextual embeddings to compare the underlying semantics, not just surface-level word overlap.
Evaluating RAG, Agents, and Workflows
To ensure high-quality generative AI applications, we evaluate three patterns differently: Retrieval-Augmented Generation, or RAG, agents, and workflows.
Evaluating a RAG system means checking retrieval relevance, meaning the system pulled the right documents; retrieval coverage, to see if any necessary info was missed; groundedness, which asks whether the answer is backed up by the retrieved content; citation accuracy, meaning the sources actually support the claims; answer relevance, to confirm the response truly answers the question; and finally, latency and cost, to ensure retrieval is fast and affordable.
Agent evaluation focuses on task completion rate, correct tool selection and usage with proper parameters, compliance with permissions, the number of steps or tool calls taken, ability to recover from a tool failure, safety of the actions, and overall cost and latency.
Workflow evaluation considers the full chain of steps. It checks that steps happen in the right order, data is passed correctly between them, retries and error handling function, humans are involved at required approval points, and the final business outcome is achieved.
While RAG evaluation emphasizes retrieval and grounding, agent evaluation centers on action-oriented criteria like tool use and safety, and workflow evaluation looks at the entire orchestrated process to ensure the business goal is met.
Q What are the key dimensions for evaluating a RAG system?
A The key dimensions are retrieval relevance, retrieval coverage, groundedness, citation accuracy, answer relevance, and latency and cost.
Q When evaluating an agent, what specifically do you check regarding tool usage?
A You check for correct tool selection, correct tool parameters, and compliance with permissions.
Q How does workflow evaluation differ from simply evaluating a single model response?
A Workflow evaluation looks at the whole process, ensuring steps are ordered correctly, data flows properly, errors are handled, human approvals happen as required, and the business outcome is achieved, rather than just checking one answer.