domain_2
Tokens
Token — a token is the smallest unit of meaning that a language model processes. It can be a whole word, part of a word, a punctuation mark, or even a single character.
How a token differs from a word. A word is one common type of token, but a token can also be just part of a word, like the 'ing' ending, or a punctuation mark like a comma. Tokens are the model's building blocks, not just whole words.
When you give a prompt to a model, it first breaks your input into tokens. The model then generates its response one token at a time.
The context window is the maximum number of tokens a model can handle in one go. It includes your input, any conversation history, and the output it generates. If the total tokens exceed this limit, the model cannot process them all.
Q What is a token in the context of generative AI?
A A token is a unit of representation, like a word, part of a word, or punctuation, that a model uses to process input and generate output.
Q How do tokens relate to a model's input and output?
A Both the prompt you provide and the response the model generates are made up of tokens. The model reads input tokens and produces output tokens one at a time.
Q What role does the context window play in token processing?
A The context window limits the total number of tokens the model can consider at once, including the input, output, and any conversation history.
Context Window
Think of a model’s context window as its working memory for one request. It’s the total amount of information the model can consider at once.
Context window — the total amount of information a model can process in a single request, including system instructions, user input, conversation history, retrieved documents, tool outputs, examples, and the text it generates.
Everything the model needs to understand your request must fit inside this window. That includes any background instructions, the chat so far, documents you pull in, and even the model’s own reply as it builds it.
A context window is not the same as a single prompt. The window holds the entire working set for one request, while a prompt is just the specific question or instruction you type in.
A larger context window lets you include more information, like a long conversation history or a big document. But it comes with a trade-off: bigger windows usually mean higher cost and slower responses.
Q What are three things that can fill up a model’s context window?
A System instructions, user input, conversation history, retrieved documents, tool outputs, examples, and generated text. Any three of those are correct.
Q Why might you choose a model with a larger context window, and what is the main downside?
A You would choose a larger context window to handle more information in one request, like a very long document or a lengthy chat. The main downside is that it increases cost and latency, meaning responses take longer and are more expensive.
Chunking
Chunking divides long documents into smaller sections for embedding vectors or Retrieval-Augmented Generation, or RAG, retrieval.
Chunks that are too large may contain irrelevant material, which reduces retrieval accuracy. The goal is to keep each chunk focused on a single idea or topic.
Chunk overlap means that consecutive chunks share some content at their boundaries. This helps preserve meaning that might otherwise be split, but it increases storage space and processing time.
To be most effective, chunking should follow meaningful boundaries such as sections, paragraphs, or topics, rather than arbitrary cutoffs.
Q What is the main purpose of chunking?
A The main purpose is to break long documents into smaller pieces so they can be turned into embedding vectors or used in RAG retrieval.
Q How does chunk size affect retrieval accuracy?
A If chunks are too large, they may include irrelevant content, which can confuse the retrieval system and lower accuracy. Ideally, chunks should be small enough to focus on a single concept.
Embeddings and Vectors
Think of an embedding as a unique fingerprint for a piece of data, like a word, sentence, or image. But instead of loops and whorls, it's a list of numbers that captures the essence of the data.
An embedding is a numerical vector that represents the meaningful characteristics of data. The numbers are chosen so that similar items have similar vectors.
In vector space, similar vectors are located near each other. So if you have a sentence about cats and another about kittens, their embeddings will be close together, even though the words are different.
Embeddings power many AI applications: semantic search, Retrieval Augmented Generation, recommendation systems, clustering, similarity search, and duplicate detection.
Latent space is a learned internal representation of the data, like a hidden layer in a neural network. It's a compressed, structured encoding that the model discovers during training, not a literal database.
Latent space differs from a raw database because it doesn't store exact data points. Instead, it captures relationships and patterns, allowing the model to generate or compare data in a meaningful way.
Q What is an embedding, and what property makes it useful?
A An embedding is a numerical vector that represents the meaningful characteristics of data. Its usefulness comes from the fact that similar items have similar vectors, so closeness in vector space reflects semantic similarity.
Q Name three applications of embeddings.
A Three applications are semantic search, Retrieval Augmented Generation, and recommendation systems. Other applications include clustering, similarity search, and duplicate detection.
Q What is latent space, and how is it different from a standard database?
A Latent space is a learned internal representation of data, like a hidden layer in a neural network. Unlike a database that stores exact data, latent space captures compressed patterns and relationships, enabling generation and comparison.
Vector Databases
A vector database is a specialized database that stores embeddings and lets you search for similar items based on meaning, not just exact words.
Embedding — a numerical representation, like a list of numbers, that captures the meaning or characteristics of data such as text, images, or audio.
Keyword search looks for exact word matches, like finding documents that contain the phrase 'annual leave'. Vector search finds items with similar meaning, so a query about requesting time off can retrieve a policy document even if the exact words differ.
Think of vector search like a librarian who understands your intent. You ask how to request vacation days, and they hand you the Annual Leave Policy, even though you never said the word 'policy'.
Amazon Web Services offers several database options that support vector storage and similarity search. These include Amazon OpenSearch Service, Amazon Aurora PostgreSQL, Amazon RDS for PostgreSQL, and Amazon Neptune.
Q What is the fundamental difference between keyword search and vector search?
A Keyword search matches exact words, while vector search finds items with similar meaning by comparing their numerical embeddings.
Q Which AWS service can you use to store embeddings and perform similarity search?
A Amazon OpenSearch Service is one option. Others include Amazon Aurora PostgreSQL, Amazon RDS for PostgreSQL, and Amazon Neptune.
Q Why would a vector database return a document titled 'Annual Leave Policy' when you search for 'How do employees request annual leave?'
A Because the vector database understands the meaning of the query and finds that the policy document's embedding is similar, even though the exact words don't match.
Transformers
A Transformer is a neural network architecture that powers many modern language and multimodal models.
Its defining feature is an attention mechanism. This mechanism evaluates how important each part of the input is in relation to every other part, allowing the model to focus on relevant context.
Because of attention, Transformers can effectively capture relationships and dependencies across long sequences of tokens, such as words in a paragraph or frames in a video.
Q What is a Transformer?
A A neural network architecture that underpins many current language and multimodal models.
Q What is the key innovation that enables Transformers to understand context so well?
A The attention mechanism, which weighs the importance of different input tokens relative to each other.
Q Why does the attention mechanism make Transformers effective with long sequences like documents?
A It can directly relate any two tokens in the sequence, regardless of their distance, so the model captures long-range dependencies and context.
Multimodal Models
Multimodal model — a generative AI model that works with more than one type of data, such as combining text with images, speech with text, or documents that mix text and tables.
These models are not stuck in one format. They can understand and generate across different kinds of input and output, bridging between, say, a picture and a written description.
Examples include describing a photo in words, creating an image from a text prompt, summarizing a video as text, or answering questions about a document that contains both paragraphs and tables.
How multimodal models differ from unimodal models: a unimodal model handles only a single data type, like text‑only generation or image‑only classification. Multimodal models combine separate generative capabilities to work across types, so they can relate, for instance, a picture and a sentence.
Q What is a multimodal model?
A A multimodal model is a generative AI model that can process and generate more than one type of data, such as text and images, speech and text, or documents with text and tables.
Q Give two examples of data types a multimodal model can combine.
A Text and images is one example. Another is speech and text. You could also have documents that mix text and tables.
Advantages of Generative AI
Generative AI has three standout advantages that make it so powerful: adaptability, responsiveness, and conversational interaction.
Adaptability — a single foundation model can handle many different tasks without being retrained, like writing emails, summarizing articles, or generating code, all from the same underlying system.
Responsiveness — the model can generate answers on the fly based on your specific instructions and the context you provide, tailoring its output to your exact needs in that moment.
Conversational interaction — the model can engage in natural, back-and-forth dialogue, remembering what you said earlier and building on the conversation just like a human would.
Adaptability is about the breadth of tasks one model can do, while responsiveness is about how well it tailors a specific answer to your immediate prompt and context.
Q What does adaptability mean in the context of generative AI?
A Adaptability means one foundation model can support many different tasks, like writing, summarizing, or coding, without needing separate models for each job.
Q How does responsiveness differ from conversational interaction?
A Responsiveness is about generating a relevant answer based on your instructions and context in a single exchange. Conversational interaction is about maintaining a natural, multi-turn dialogue where the model remembers and builds on the history.
Q Why is conversational interaction an advantage for users?
A It allows for a natural, human-like dialogue where you can ask follow-up questions, clarify points, and explore ideas step by step, making the interaction more intuitive and effective.