{ "cells": [ { "cell_type": "markdown", "id": "6af1390c", "metadata": {}, "source": [ "# What Is an LLM? A Mental Model for Software Engineers\n", "\n", "An original Beyond the Stack article companion. The numbered sections match the planned article.\n", "Follow a synthetic delivery update from text to tokens, numerical representations, and generated text.\n", "The article is planned; no publication URL exists yet.\n", "\n", "**Run:** select the environment described in the README, restart the kernel, and run all cells in order.\n", "The first run downloads approximately 1 GB of Qwen weights plus tokenizer files. CPU inference uses\n", "additional memory (float32 weights alone are about 2 GB). There are no paid API calls or training steps.\n", "W&B logging is optional and offline; uploading would require a separate deliberate action.\n", "Saved outputs are evidence of a particular run, not a promise of identical wording or speed elsewhere." ] }, { "cell_type": "markdown", "id": "bae222b7-acd1-457a-9621-5ed8eb488911", "metadata": {}, "source": [ "## High-level overview of LLM" ] }, { "cell_type": "markdown", "id": "9cb7fc9b-c847-403a-a00c-cebd966329d7", "metadata": { "jp-MarkdownHeadingCollapsed": true }, "source": [ "### Prefill\n", "\n", "1. The application formats the system and user messages using Qwen’s chat template. In our example, the result contains **68 token IDs**: the system message, delivery prompt, role markers, and opening assistant marker. The delivery prompt alone contains 49 tokens.\n", "\n", "2. Each ID selects a **896-value initial embedding** from the model’s learned input embedding table. We now have one vector for each input position. The same token ID can appear at several positions.\n", "\n", "3. The 68 representations pass through **24 transformer blocks in sequence**. Each block creates its own queries, keys, and values (**Q/K/V**) from the representations it receives. It uses them in attention, applies its other operations, and sends updated representations to the next block. Block 1 works from the initial embeddings; later blocks work from representations updated by earlier blocks.\n", "\n", "4. This model uses **causal attention**: each position can use itself and earlier positions, but cannot use later positions. The first position is still updated because it can attend to itself and undergo the block’s other operations. The final input position can use the entire supplied prefix. This restriction supports **causal language modeling**, in which the model predicts the next token from the prefix available so far. Other model setups include **masked language modeling**, which predicts hidden tokens using surrounding text, and **encoder–decoder models**, which generate an output using a separately processed input.\n", "\n", "5. After block 24, there is one final 896-value representation for each of the 68 input positions. This **first forward pass** through the complete supplied input is called **prefill**. During it, each block also produces keys and values for its positions that can be cached for later generation steps.\n", "\n", "### Overall flow: choosing the first answer token\n", "\n", "1. **Format and tokenize:** Convert the system and user messages into the 68 token IDs the model expects.\n", "\n", "2. **Look up embeddings:** Obtain 68 initial vectors, each with 896 values, from the trained embedding table.\n", "\n", "3. **Process the input:** Pass those representations through all 24 transformer blocks. Each block creates and uses Q/K/V and updates the representations. This is the prefill pass.\n", "\n", "4. **Take the last position’s final vector:** Select its 896-value representation. Because of causal attention, that position has access to the whole supplied prefix.\n", "\n", "5. **Score possible next tokens:** The model’s final output layer, **`lm_head`**, transforms that vector into **151,936 logits**. It is a linear layer shown in the notebook as `Linear(in_features=896, out_features=151936, bias=False)`. In simplified notation:\n", "\n", " ```text\n", " logits = W × final_vector\n", " ```\n", "\n", " Here, `W` is the output layer’s learned matrix. The result contains one logit for each numbered output position associated with a possible token ID. These positions are sometimes informally called *output slots*; **vocabulary size** is the count of them, not a synonym for one slot. The model’s configured output size is 151,936, which can be larger than the tokenizer’s count of assigned tokens. The layer scores the full output before the notebook displays only the leading candidates.\n", "\n", "6. **Choose one token ID:** Apply a **decoding strategy** to the logits. Greedy decoding chooses the highest-scoring ID. Sampling uses softmax to make a probability distribution and draws an ID; **temperature** changes how concentrated that distribution is. Other widely used selection methods include *top-k* and *top-p* sampling, which restrict the candidates considered for a draw, and *beam search*, which tracks several possible continuations. The notebook focuses on greedy decoding and sampling.\n", "\n", " **Scoring** is the model’s calculation of logits; **choosing** is the decoding strategy applied afterward. The notebook also calculates softmax probabilities for inspection, although greedy selection can use the logits directly. A next-token probability does not establish that the answer is factually correct.\n", "\n", "7. **Append the selected ID:** This is answer token 1. The logits that led to it came from prefill. The token itself has **not yet passed through the transformer blocks**; that happens in the next pass, when the model predicts answer token 2.\n", "\n", "### The generation loop (prefill-decoding-decoding-...-end of response token)\n", "\n", "```text\n", "Prefill: process all 68 input positions through blocks 1–24\n", " → score possible next tokens using the last input position\n", " → choose and append answer token 1\n", "\n", "Decode step: look up answer token 1’s initial embedding\n", " → pass its position through blocks 1–24\n", " → in each block, create Q/K/V for this new position\n", " and use that block’s cached keys and values from earlier positions\n", " → score possible next tokens using its final representation\n", " → choose and append answer token 2\n", "\n", "Decode step: process answer token 2 in the same way\n", " → score, choose, and append answer token 3\n", "\n", "Repeat decode steps until an end-of-response token is selected\n", "or a configured generation limit is reached.\n", "```\n", "\n", "Both **prefill** and a **decode step** use the embedding lookup and transformer blocks. Prefill processes the supplied input positions. With the cache enabled, each subsequent decode step processes the newly appended position and reuses keys and values saved for earlier positions.\n", "\n", "The new token starts at block 1 with its **own initial embedding**. It does not start with the earlier input’s final block-24 representations. Each block uses **its own cached keys and values** from prefill and preceding decode steps.\n", "\n", "### Ending the response\n", "\n", "The model can select a special **end-of-response token** when it predicts the next token. The generation code recognizes that token ID and stops. The model learned when such a token tends to occur, but it may stop too early or continue longer than intended.\n", "\n", "A configured generation limit provides another stopping condition. In this notebook, `MAX_NEW_TOKENS = 64`, so generation stops after at most 64 new tokens even if the model has not selected an end-of-response token. Reaching that limit does not mean the answer is complete." ] }, { "cell_type": "markdown", "id": "23acaa58", "metadata": {}, "source": [ "## 1. You send a prompt. What actually happens?\n", "\n", "We want a customer-facing sentence based only on the supplied update. How does text become\n", "model input? Where did the model's ability to write come from? How is the answer produced?\n", "We will return to an actual generated response in Section 5, after inspecting the input.\n", "This is synthetic data; no customer records or notification service are involved." ] }, { "cell_type": "code", "execution_count": 1, "id": "8de26288", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Summarize this delivery update in one sentence to notify the customer:\n", "“Order 482 was scheduled for 18:00. The driver reports a van breakdown. The revised arrival time is 18:25.”\n" ] } ], "source": [ "PROMPT = (\n", " 'Summarize this delivery update in one sentence to notify the customer:\\n'\n", " '“Order 482 was scheduled for 18:00. The driver reports a van breakdown. '\n", " 'The revised arrival time is 18:25.”'\n", ")\n", "print(PROMPT)" ] }, { "cell_type": "markdown", "id": "9c02ca1d", "metadata": {}, "source": [ "## 2. What is an LLM, and where do its abilities come from?\n", "\n", "A model is a neural network that performs numerical operations using learned **parameters**.\n", "During pretraining, a causal language model is trained to predict subsequent tokens: an optimisation\n", "procedure adjusts parameters to reduce prediction error across training examples.\n", "Instruction tuning uses examples of instructions and desired responses to improve instruction following;\n", "other post-training methods may follow. These stages differ from **inference**: using the trained model.\n", "\n", "We load an instruction-tuned model, turn off training behaviour with `eval()`, and disable gradient\n", "tracking. We do not train, update weights, or call an external generation API in this notebook.\n", "`eval()` alone does not disable gradients; `inference_mode()` below does that for the forward passes.\n", "\n", "- [Causal language modeling — Hugging Face](https://huggingface.co/docs/transformers/v4.57.1/en/tasks/language_modeling)\n", "- [Instruction following with human feedback — Ouyang et al., 2022](https://arxiv.org/abs/2203.02155)\n", "- [Selected model card — Qwen team](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)" ] }, { "cell_type": "code", "execution_count": 2, "id": "4c7ad621", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
packageversion
0torch2.8.0
1transformers4.57.6
2tokenizers0.22.2
3huggingface-hub0.36.2
4safetensors0.7.0
5pandas2.3.3
6matplotlib3.9.4
7nbformat5.10.4
8ipykernel6.31.0
\n", "
" ], "text/plain": [ " package version\n", "0 torch 2.8.0\n", "1 transformers 4.57.6\n", "2 tokenizers 0.22.2\n", "3 huggingface-hub 0.36.2\n", "4 safetensors 0.7.0\n", "5 pandas 2.3.3\n", "6 matplotlib 3.9.4\n", "7 nbformat 5.10.4\n", "8 ipykernel 6.31.0" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "Qwen2ForCausalLM(\n", " (model): Qwen2Model(\n", " (embed_tokens): Embedding(151936, 896)\n", " (layers): ModuleList(\n", " (0-23): 24 x Qwen2DecoderLayer(\n", " (self_attn): Qwen2Attention(\n", " (q_proj): Linear(in_features=896, out_features=896, bias=True)\n", " (k_proj): Linear(in_features=896, out_features=128, bias=True)\n", " (v_proj): Linear(in_features=896, out_features=128, bias=True)\n", " (o_proj): Linear(in_features=896, out_features=896, bias=False)\n", " )\n", " (mlp): Qwen2MLP(\n", " (gate_proj): Linear(in_features=896, out_features=4864, bias=False)\n", " (up_proj): Linear(in_features=896, out_features=4864, bias=False)\n", " (down_proj): Linear(in_features=4864, out_features=896, bias=False)\n", " (act_fn): SiLUActivation()\n", " )\n", " (input_layernorm): Qwen2RMSNorm((896,), eps=1e-06)\n", " (post_attention_layernorm): Qwen2RMSNorm((896,), eps=1e-06)\n", " )\n", " )\n", " (norm): Qwen2RMSNorm((896,), eps=1e-06)\n", " (rotary_emb): Qwen2RotaryEmbedding()\n", " )\n", " (lm_head): Linear(in_features=896, out_features=151936, bias=False)\n", ")" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import importlib.metadata as metadata\n", "import json\n", "import os\n", "import platform\n", "import time\n", "import warnings\n", "from datetime import datetime, timezone\n", "from pathlib import Path\n", "\n", "# Suppress two known environment notices; no package changes are needed for this demo.\n", "warnings.filterwarnings('ignore', message='urllib3 v2 only supports OpenSSL.*')\n", "warnings.filterwarnings('ignore', message='IProgress not found.*')\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n", "import torch\n", "from IPython.display import display\n", "from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig\n", "\n", "MODEL_ID = 'Qwen/Qwen2.5-0.5B-Instruct'\n", "MODEL_REVISION = '7ae557604adf67be50417f59c2c2f167def9a775'\n", "DEVICE = 'cpu' # Portable teaching baseline; timings are not production benchmarks.\n", "DTYPE = torch.float32 # Store floating-point model weights as 32-bit numbers.\n", "torch.set_num_threads(2)\n", "MAX_NEW_TOKENS = 64\n", "TRACE_STEPS = 8\n", "SEEDS = [7, 19]\n", "LOG_WANDB = False # Optional: change to True to record tables offline in Section 7.\n", "\n", "versions = {name: metadata.version(name) for name in (\n", " 'torch', 'transformers', 'tokenizers', 'huggingface-hub',\n", " 'safetensors', 'pandas', 'matplotlib', 'nbformat', 'ipykernel'\n", ")}\n", "manifest = {\n", " 'model_id': MODEL_ID, 'revision': MODEL_REVISION,\n", " 'python': platform.python_version(), 'platform': platform.platform(),\n", " 'device': DEVICE, 'dtype': str(DTYPE), 'torch_threads': torch.get_num_threads(),\n", " 'versions': versions, 'started_at_utc': datetime.now(timezone.utc).isoformat(),\n", "}\n", "display(pd.DataFrame(versions.items(), columns=['package', 'version']))\n", "\n", "# Default Hugging Face cache; HF_HUB_CACHE can override its location before launch.\n", "# On a fresh machine this downloads weights. Later runs can reuse the cache.\n", "# Select the causal language model architecture and load its pretrained configuration and weights.\n", "model = AutoModelForCausalLM.from_pretrained(\n", " MODEL_ID, revision=MODEL_REVISION, dtype=DTYPE,\n", " attn_implementation='eager', # Explicit, inspectable baseline.\n", ").to(DEVICE)\n", "model.eval() # Use evaluation behavior, such as disabling dropout where present.\n", "model.requires_grad_(False) # Do not calculate gradients for model parameters." ] }, { "cell_type": "code", "execution_count": 3, "id": "7da1c175", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
propertyvalue
0architectureqwen2
1parameters494032768
2transformer blocks24
3representation width896
4input embedding shape / vocabulary size(151936, 896)
5training modeFalse
\n", "
" ], "text/plain": [ " property value\n", "0 architecture qwen2\n", "1 parameters 494032768\n", "2 transformer blocks 24\n", "3 representation width 896\n", "4 input embedding shape / vocabulary size (151936, 896)\n", "5 training mode False" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "config_summary = pd.DataFrame([\n", " ('architecture', model.config.model_type),\n", " ('parameters', sum(p.numel() for p in model.parameters())),\n", " ('transformer blocks', model.config.num_hidden_layers),\n", " ('representation width', model.config.hidden_size),\n", " ('input embedding shape / vocabulary size', tuple(model.get_input_embeddings().weight.shape)), # \n", " ('training mode', model.training),\n", "], columns=['property', 'value'])\n", "display(config_summary)\n", "assert not model.training\n", "assert not any(p.requires_grad for p in model.parameters())\n", "\n" ] }, { "cell_type": "markdown", "id": "03874c26", "metadata": {}, "source": [ "These are real properties of the loaded model, not measurements of intelligence or reliability.\n", "The parameters carry learned patterns; this prompt provides the specific delivery facts for this request.\n", "The model has not looked up a live order or learned a new delivery policy from its identifier.\n", "The model printout is a map of its components, not a generated answer. `Embedding(151936, 896)`\n", "means the input embedding table has 151,936 rows, each containing 896 learned values. A token ID\n", "selects a row. The tokenizer vocabulary is its set of distinct tokens, and its vocabulary size is the\n", "number of those tokens. The model's configured vocabulary size (151,936) gives the embedding table\n", "capacity; it can be larger than the tokenizer's actual vocabulary size. This count describes token\n", "types, not the number of token positions that fit in the context window.\n", "The model has 24 decoder blocks. Each block includes attention and an MLP (another transformation).\n", "The final `lm_head` produces scores for possible next tokens.\n", "\n", "Each block has 14 query heads and 2 key/value heads, each 64 values wide. Several query heads\n", "share each key/value head; this is called grouped-query attention. `q_proj`, `k_proj`, and `v_proj`\n", "are learned layers that create query, key, and value vectors from a position's current representation.\n", "Attention uses queries and keys to weight information from the values at permitted positions.\n", "\n", "`model.eval()` selects evaluation behavior; it does not disable gradient tracking by itself.\n", "Gradients tell a training algorithm how changing each parameter would affect its objective.\n", "`requires_grad_(False)` marks model parameters as not needing those calculations. The later\n", "`torch.inference_mode()` blocks also disable gradient tracking during inference. None of these\n", "settings trains the model or changes its learned weights.\n" ] }, { "cell_type": "markdown", "id": "2abdaccf", "metadata": {}, "source": [ "## 3. How text becomes model input (prefill)\n", "\n", "Use the tokenizer shipped with the model: its token IDs must match the model's embedding rows.\n", "Its vocabulary size counts distinct tokens, including special tokens, not tokens in a particular prompt.\n", "Qwen uses byte-level BPE; the tokenization algorithm itself is a follow-up topic.\n", "Token IDs are lookup indices, not meanings, probabilities, or a ranking of words.\n", "\n", "First inspect the prompt on its own. Then inspect the actual conversation format: the model's\n", "**chat template** adds role boundaries and an assistant prefix. We explicitly include a generic\n", "system instruction so the input is visible and controlled, rather than relying on a template default.\n", "\n", "- [Qwen tokenization and chat templates](https://qwen.readthedocs.io/en/latest/getting_started/concepts.html)\n", "- [Hugging Face chat templates](https://huggingface.co/docs/transformers/v4.57.1/en/chat_templating)" ] }, { "cell_type": "code", "execution_count": 30, "id": "3106b503", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
positiontoken_idvocabulary_piecedecoded_piecespecial_token
009190Sum'Sum'False
115612mar'mar'False
22551ize'ize'False
33419Ġthis' this'False
449691Ġdelivery' delivery'False
552647Ġupdate' update'False
66304Ġin' in'False
77825Ġone' one'False
8811652Ġsentence' sentence'False
99311Ġto' to'False
101015456Ġnotify' notify'False
1111279Ġthe' the'False
12126002Ġcustomer' customer'False
1313510:Ċ':\\n'False
14142073âĢľ'“'False
15154431Order'Order'False
1616220Ġ' 'False
1717194'4'False
1818238'8'False
1919172'2'False
2020572Ġwas' was'False
212113537Ġscheduled' scheduled'False
2222369Ġfor' for'False
2323220Ġ' 'False
2424161'1'False
2525238'8'False
262625:':'False
2727150'0'False
2828150'0'False
292913.'.'False
3030576ĠThe' The'False
31315579Ġdriver' driver'False
32326682Ġreports' reports'False
3333264Ġa' a'False
34345242Ġvan' van'False
353529985Ġbreakdown' breakdown'False
363613.'.'False
3737576ĠThe' The'False
383831728Ġrevised' revised'False
393918647Ġarrival' arrival'False
4040882Ġtime' time'False
4141374Ġis' is'False
4242220Ġ' 'False
4343161'1'False
4444238'8'False
454525:':'False
4646172'2'False
4747205'5'False
48481987.âĢĿ'.”'False
\n", "
" ], "text/plain": [ " position token_id vocabulary_piece decoded_piece special_token\n", "0 0 9190 Sum 'Sum' False\n", "1 1 5612 mar 'mar' False\n", "2 2 551 ize 'ize' False\n", "3 3 419 Ġthis ' this' False\n", "4 4 9691 Ġdelivery ' delivery' False\n", "5 5 2647 Ġupdate ' update' False\n", "6 6 304 Ġin ' in' False\n", "7 7 825 Ġone ' one' False\n", "8 8 11652 Ġsentence ' sentence' False\n", "9 9 311 Ġto ' to' False\n", "10 10 15456 Ġnotify ' notify' False\n", "11 11 279 Ġthe ' the' False\n", "12 12 6002 Ġcustomer ' customer' False\n", "13 13 510 :Ċ ':\\n' False\n", "14 14 2073 âĢľ '“' False\n", "15 15 4431 Order 'Order' False\n", "16 16 220 Ġ ' ' False\n", "17 17 19 4 '4' False\n", "18 18 23 8 '8' False\n", "19 19 17 2 '2' False\n", "20 20 572 Ġwas ' was' False\n", "21 21 13537 Ġscheduled ' scheduled' False\n", "22 22 369 Ġfor ' for' False\n", "23 23 220 Ġ ' ' False\n", "24 24 16 1 '1' False\n", "25 25 23 8 '8' False\n", "26 26 25 : ':' False\n", "27 27 15 0 '0' False\n", "28 28 15 0 '0' False\n", "29 29 13 . '.' False\n", "30 30 576 ĠThe ' The' False\n", "31 31 5579 Ġdriver ' driver' False\n", "32 32 6682 Ġreports ' reports' False\n", "33 33 264 Ġa ' a' False\n", "34 34 5242 Ġvan ' van' False\n", "35 35 29985 Ġbreakdown ' breakdown' False\n", "36 36 13 . '.' False\n", "37 37 576 ĠThe ' The' False\n", "38 38 31728 Ġrevised ' revised' False\n", "39 39 18647 Ġarrival ' arrival' False\n", "40 40 882 Ġtime ' time' False\n", "41 41 374 Ġis ' is' False\n", "42 42 220 Ġ ' ' False\n", "43 43 16 1 '1' False\n", "44 44 23 8 '8' False\n", "45 45 25 : ':' False\n", "46 46 17 2 '2' False\n", "47 47 20 5 '5' False\n", "48 48 1987 .âĢĿ '.”' False" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Prompt token count: 49\n", "Tokenizer vocabulary size: 151665\n" ] } ], "source": [ "tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, revision=MODEL_REVISION)\n", "\n", "def token_table(ids):\n", " return pd.DataFrame([\n", " {\n", " 'position': position,\n", " 'token_id': token_id,\n", " 'vocabulary_piece': tokenizer.convert_ids_to_tokens(token_id),\n", " 'decoded_piece': repr(tokenizer.decode([token_id])),\n", " 'special_token': token_id in tokenizer.all_special_ids,\n", " }\n", " for position, token_id in enumerate(ids)\n", " ])\n", "\n", "raw_ids = tokenizer.encode(PROMPT, add_special_tokens=False)\n", "raw_tokens = token_table(raw_ids)\n", "display(raw_tokens)\n", "print('Prompt token count:', len(raw_ids))\n", "assert tokenizer.decode(raw_ids) == PROMPT\n", "print(\"Tokenizer vocabulary size:\", len(tokenizer))" ] }, { "cell_type": "markdown", "id": "0eb645cf", "metadata": {}, "source": [ "The vocabulary spelling can contain space or byte markers. Also, decoding one byte-level token\n", "alone can show a replacement character when its bytes form only part of a Unicode character.\n", "That is why the full-sequence round-trip check matters; isolated pieces need not look like words." ] }, { "cell_type": "code", "execution_count": 5, "id": "6a8ea313", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Actual formatted input:\n", "\n", "<|im_start|>system\n", "You are a helpful assistant.<|im_end|>\n", "<|im_start|>user\n", "Summarize this delivery update in one sentence to notify the customer:\n", "“Order 482 was scheduled for 18:00. The driver reports a van breakdown. The revised arrival time is 18:25.”<|im_end|>\n", "<|im_start|>assistant\n", "\n", "Prompt alone: 49 tokens\n", "Complete model input: 68 tokens\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
positiontoken_idvocabulary_piecedecoded_piecespecial_token
00151644<|im_start|>'<|im_start|>'True
118948system'system'False
22198Ċ'\\n'False
332610You'You'False
44525Ġare' are'False
55264Ġa' a'False
6610950Ġhelpful' helpful'False
7717847Ġassistant' assistant'False
8813.'.'False
99151645<|im_end|>'<|im_end|>'True
1010198Ċ'\\n'False
1111151644<|im_start|>'<|im_start|>'True
1212872user'user'False
1313198Ċ'\\n'False
14149190Sum'Sum'False
15155612mar'mar'False
1616551ize'ize'False
1717419Ġthis' this'False
\n", "
" ], "text/plain": [ " position token_id vocabulary_piece decoded_piece special_token\n", "0 0 151644 <|im_start|> '<|im_start|>' True\n", "1 1 8948 system 'system' False\n", "2 2 198 Ċ '\\n' False\n", "3 3 2610 You 'You' False\n", "4 4 525 Ġare ' are' False\n", "5 5 264 Ġa ' a' False\n", "6 6 10950 Ġhelpful ' helpful' False\n", "7 7 17847 Ġassistant ' assistant' False\n", "8 8 13 . '.' False\n", "9 9 151645 <|im_end|> '<|im_end|>' True\n", "10 10 198 Ċ '\\n' False\n", "11 11 151644 <|im_start|> '<|im_start|>' True\n", "12 12 872 user 'user' False\n", "13 13 198 Ċ '\\n' False\n", "14 14 9190 Sum 'Sum' False\n", "15 15 5612 mar 'mar' False\n", "16 16 551 ize 'ize' False\n", "17 17 419 Ġthis ' this' False" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "MESSAGES = [\n", " {'role': 'system', 'content': 'You are a helpful assistant.'},\n", " {'role': 'user', 'content': PROMPT},\n", "]\n", "# readable form\n", "formatted_prompt = tokenizer.apply_chat_template(\n", " MESSAGES, tokenize=False, add_generation_prompt=True,\n", ")\n", "\n", "# not much readable - the actual tokens to go in a model's input\n", "inputs = tokenizer.apply_chat_template(\n", " MESSAGES, tokenize=True, add_generation_prompt=True,\n", " return_dict=True, return_tensors='pt',\n", ").to(DEVICE)\n", "\n", "input_ids = inputs['input_ids']\n", "input_length = input_ids.shape[1]\n", "print('Actual formatted input:\\n')\n", "print(formatted_prompt)\n", "print('Prompt alone:', len(raw_ids), 'tokens')\n", "print('Complete model input:', input_length, 'tokens')\n", "display(token_table(input_ids[0].tolist()).head(18))\n", "assert tokenizer.encode(formatted_prompt, add_special_tokens=False) == input_ids[0].tolist()" ] }, { "cell_type": "markdown", "id": "6a24e515-954f-4c68-81c2-ab55ecd2a243", "metadata": {}, "source": [ "### Getting the initial **embeddings vector** for input tokens (prefill)" ] }, { "cell_type": "code", "execution_count": 8, "id": "840cef8a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Token IDs shape: (1, 68) (batch, positions)\n", "Embedding shape: (1, 68, 896) (batch, positions, width)\n", "First token, first 8 embedding values: [0.0013885498046875, -0.0084228515625, 0.00732421875, -0.0015869140625, -0.0079345703125, 0.00494384765625, -0.007659912109375, 0.01263427734375]\n" ] } ], "source": [ "with torch.inference_mode():\n", " embedding_table = model.get_input_embeddings() # Access the model's input embedding layer - from training\n", " input_embeddings = embedding_table(input_ids) # Look up a vector for each input token ID.\n", " \n", "print('Token IDs shape:', tuple(input_ids.shape), '(batch, positions)')\n", "print('Embedding shape:', tuple(input_embeddings.shape), '(batch, positions, width)')\n", "print('First token, first 8 embedding values:', input_embeddings[0, 0, :8].tolist())\n", "assert input_embeddings.shape[:2] == input_ids.shape" ] }, { "cell_type": "markdown", "id": "9092de9e", "metadata": {}, "source": [ "An embedding is a learned numerical representation, looked up here using the token ID.\n", "The input embedding table has one row per model vocabulary slot; the tensor displayed above instead\n", "has one representation per input position. Repeated token IDs can occupy multiple input positions.\n", "The few displayed coordinates do not individually label concepts such as “delay” or “customer”.\n", "Qwen also supplies position information through rotary position embeddings inside attention;\n", "the lookup tensor above is not the complete position-aware processing of the prompt." ] }, { "cell_type": "markdown", "id": "f28250f2", "metadata": {}, "source": [ "## 4. How the model uses the surrounding text (prefill - **attention calculation**)\n", "\n", "A **position** is a token's place in the input sequence. The same token ID can occur at multiple\n", "positions. Its initial embedding is the same lookup at each occurrence, but the representations can\n", "change as the model processes their different contexts. (attention generation)\n", "\n", "The model sends these representations through 24 repeated computational blocks. **Attention** is\n", "an operation within each block that combines information from permitted positions; other operations\n", "also transform the representations. A **transformer** is the architecture organising these blocks.\n", "Qwen incorporates position information inside attention, so order matters too.\n", "\n", "In this **causal** model, each position can use information from itself and earlier positions, not\n", "later ones. For example, position 3 can use positions 1, 2 and 3. The final input position can draw\n", "on the entire supplied prefix, including the instruction, order number, cause, and revised time.\n", "\n", "After the blocks, we have one updated 896-value representation for each of the 68 input positions:\n", "68 vectors, not 896 vectors. Each position's representation reflects only its permitted context.\n", "The final position's representation is used to produce scores for the first answer token; once a\n", "token is selected and appended, it becomes part of the context for generating the next token.\n", "\n", "The returned **hidden states** are the numerical representations at the embedding stage and across\n", "blocks. We inspect their shapes and a numerical summary, without claiming that a particular\n", "coordinate, block, or attention head explains the answer. Attention equations and the details of\n", "heads and Q/K/V are deferred to a separate article.\n", "\n", "[Original Transformer paper — Vaswani et al., 2017](https://arxiv.org/abs/1706.03762)\n" ] }, { "cell_type": "code", "execution_count": 11, "id": "873216d9", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
stageshapelast_position_norm
0embedding lookup (initial model's emebeddings)(1, 68, 896)0.579821
1after block/layer 1(1, 68, 896)3.193251
23after block/layer 23(1, 68, 896)66.746880
24final normalised state(1, 68, 896)301.364136
\n", "
" ], "text/plain": [ " stage shape \\\n", "0 embedding lookup (initial model's emebeddings) (1, 68, 896) \n", "1 after block/layer 1 (1, 68, 896) \n", "23 after block/layer 23 (1, 68, 896) \n", "24 final normalised state (1, 68, 896) \n", "\n", " last_position_norm \n", "0 0.579821 \n", "1 3.193251 \n", "23 66.746880 \n", "24 301.364136 " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "torch.Size([1, 68])\n", "Output scores shape: (1, 68, 151936) (batch, positions, vocabulary)\n" ] } ], "source": [ "with torch.inference_mode():\n", " inspected = model(**inputs, output_hidden_states=True, use_cache=False)\n", "\n", "# There is one initial embedding tensor plus one returned tensor per block.\n", "# Qwen's final returned state includes its final normalisation.\n", "state_rows = []\n", "for index, state in enumerate(inspected.hidden_states):\n", " label = (\"embedding lookup (initial model's emebeddings)\" if index == 0 else\n", " 'final normalised state' if index == model.config.num_hidden_layers else\n", " f'after block/layer {index}')\n", " state_rows.append({\n", " 'stage': label, 'shape': str(tuple(state.shape)),\n", " 'last_position_norm': state[0, -1].float().norm().item(), # converting last token's 896 vector to single value - to show it is different from previous block\n", " })\n", "display(pd.DataFrame(state_rows).iloc[[0, 1, -2, -1]])\n", "print(inspected.logits.shape[:2])\n", "print('Output scores shape:', tuple(inspected.logits.shape), '(batch, positions, vocabulary)')\n", "assert len(inspected.hidden_states) == model.config.num_hidden_layers + 1" ] }, { "cell_type": "markdown", "id": "714dde7c", "metadata": {}, "source": [ "Each `state` is a tensor with shape `(1, 68, 896)`: one input sequence (could be more than 1 i.e. can process batch of messages/inputs), 68 token positions,\n", "and 896 values per position. The one sequence is the complete formatted conversation, including\n", "the system message, delivery prompt, role markers, and assistant prefix. The table shows three of\n", "25 returned states: the initial embedding lookup, the state after block 1, and the final state after\n", "block 24 and final normalization. State 0 is the lookup from the trained embedding table, before\n", "attention. Inference does not update the table.\n", "\n", "`state[0, -1]` selects the first (and only) sequence with `0`, then its last token position with\n", "`-1`. It keeps all 896 values, so the result has shape `(896,)`. Python indices start at zero;\n", "negative indices count from the end. At the final stage, that position's representation can draw\n", "on the whole input prefix.\n", "\n", "`last_position_norm` is the L2 norm of that 896-value vector: the square root of the sum of its\n", "squared values. It is a single number describing the vector's magnitude. It is not a measure of\n", "understanding or answer quality, and different vectors can have the same norm. The table shows\n", "shapes and this limited numerical summary, not the full vectors or a complete comparison of them.\n", "\n", "The model produces scores at every supplied position. For generation, we use the scores at the\n", "**last position** to select the next token.\n" ] }, { "cell_type": "markdown", "id": "63470d9c", "metadata": {}, "source": [ "## 5. How an answer is generated (decoding step)\n", "\n", "At each step the model returns **logits** (unnormalised scores) for vocabulary tokens.\n", "**Softmax** turns these into a probability distribution. A decoder selects a token, appends it,\n", "and repeats. Greedy decoding takes the highest score; sampling draws from a distribution.\n", "Temperature rescales logits before sampling. None of these probabilities establishes factual truth.\n", "\n", "We first expose eight real greedy steps. The full prompt is processed on the first pass\n", "(prefill). Later passes process the newly selected token and reuse cached prior key/value states.\n", "The cache is a computation optimisation, not conversation memory stored across independent requests.\n", "\n", "[Transformers generation reference](https://huggingface.co/docs/transformers/v4.57.1/en/main_classes/text_generation)\n", "and [caching explanation](https://huggingface.co/docs/transformers/v4.57.1/en/cache_explanation)." ] }, { "cell_type": "code", "execution_count": 15, "id": "9df31f62-3199-44e4-a558-a4667d1c6f31", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[151645, 151643]\n" ] } ], "source": [ "eos_config = model.generation_config.eos_token_id # end-of-sequence token ID\n", "print(eos_config)" ] }, { "cell_type": "code", "execution_count": 12, "id": "c74e018a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
stepranktoken_idpieceprobability
011785'The'0.298447
1124431'Order'0.103310
21332'A'0.094836
3145501'Please'0.068357
41538121'Delivery'0.062146
5219691' delivery'0.243475
6226002' customer'0.202478
7235579' driver'0.181432
8241973' order'0.134751
925220' '0.039878
\n", "
" ], "text/plain": [ " step rank token_id piece probability\n", "0 1 1 785 'The' 0.298447\n", "1 1 2 4431 'Order' 0.103310\n", "2 1 3 32 'A' 0.094836\n", "3 1 4 5501 'Please' 0.068357\n", "4 1 5 38121 'Delivery' 0.062146\n", "5 2 1 9691 ' delivery' 0.243475\n", "6 2 2 6002 ' customer' 0.202478\n", "7 2 3 5579 ' driver' 0.181432\n", "8 2 4 1973 ' order' 0.134751\n", "9 2 5 220 ' ' 0.039878" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
stepselected_idselected_piecegenerated_so_far
01785'The'The
129691' delivery'The delivery
23572' was'The delivery was
3413537' scheduled'The delivery was scheduled
45369' for'The delivery was scheduled for
56220' 'The delivery was scheduled for
6716'1'The delivery was scheduled for 1
7823'8'The delivery was scheduled for 18
\n", "
" ], "text/plain": [ " step selected_id selected_piece generated_so_far\n", "0 1 785 'The' The\n", "1 2 9691 ' delivery' The delivery\n", "2 3 572 ' was' The delivery was\n", "3 4 13537 ' scheduled' The delivery was scheduled\n", "4 5 369 ' for' The delivery was scheduled for\n", "5 6 220 ' ' The delivery was scheduled for \n", "6 7 16 '1' The delivery was scheduled for 1\n", "7 8 23 '8' The delivery was scheduled for 18" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "eos_config = model.generation_config.eos_token_id # end-of-sequence token ID\n", "EOS_IDS = set(eos_config if isinstance(eos_config, list) else [eos_config])\n", "PAD_ID = tokenizer.pad_token_id # Padding ID: Padding exists when you process several sequences together in a batch and they have different lengths\n", "if PAD_ID is None:\n", " PAD_ID = next(iter(EOS_IDS))\n", "\n", "selected_ids = []\n", "candidate_token_rows = []\n", "decoded_step_rows = []\n", "probability_sums = []\n", "past = None\n", "step_input = input_ids\n", "attention_mask = inputs['attention_mask'].clone()\n", "first_logits = None\n", "\n", "with torch.inference_mode():\n", " for step in range(TRACE_STEPS):\n", " output = model(\n", " input_ids=step_input, attention_mask=attention_mask,\n", " past_key_values=past, use_cache=True,\n", " )\n", " logits = output.logits[0, -1].float()\n", " if first_logits is None:\n", " first_logits = logits.clone()\n", " probabilities = torch.softmax(logits.double(), dim=-1)\n", " probability_sums.append(probabilities.sum().item())\n", " values, candidates = probabilities.topk(5)\n", " chosen = int(logits.argmax())\n", " \n", " # the candidate tokens with softmax (from logit) and ranking\n", " for rank, (candidate, probability) in enumerate(zip(candidates.tolist(), values.tolist()), 1):\n", " candidate_token_rows.append({\n", " 'step': step + 1, 'rank': rank, 'token_id': candidate,\n", " 'piece': repr(tokenizer.decode([candidate])),\n", " 'probability': probability,\n", " })\n", " selected_ids.append(chosen)\n", "\n", " # result of each decoding step (8 here) \n", " decoded_step_rows.append({\n", " 'step': step + 1, 'selected_id': chosen,\n", " 'selected_piece': repr(tokenizer.decode([chosen])),\n", " 'generated_so_far': tokenizer.decode(selected_ids, skip_special_tokens=False),\n", " })\n", " past = output.past_key_values\n", " if chosen in EOS_IDS:\n", " break\n", " step_input = torch.tensor([[chosen]], device=DEVICE)\n", " attention_mask = torch.cat([\n", " attention_mask, attention_mask.new_ones((1, 1)),\n", " ], dim=1)\n", "\n", "candidates_df = pd.DataFrame(candidate_token_rows) # the candidate tokens with softmax (from logit) and ranking\n", "steps_df = pd.DataFrame(decoded_step_rows) # result of each decoding step (8 here) \n", "display(candidates_df[candidates_df['step'] <= 2])\n", "display(steps_df)\n", "assert all(abs(total - 1.0) < 1e-5 for total in probability_sums)\n", "del past, output" ] }, { "cell_type": "code", "execution_count": 13, "id": "a8811c58", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAArEAAAEiCAYAAADuwIpdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8ekN5oAAAACXBIWXMAAA9hAAAPYQGoP6dpAABAEklEQVR4nO3dCZxN9f/H8e/Yxr5ll6zJki1CWhBCikKIirKlRSJbZY0Q2kSJUJKlklQqS2mRkrKFyr6UEsnY1/N/vL//37mPe+/cmbnDzFxn5vV8PG6555577vd8z/Y5n/P9fifKcRzHAAAAAB6SLtIFAAAAABKLIBYAAACeQxALAAAAzyGIBQAAgOcQxAIAAMBzCGIBAADgOQSxAAAA8ByCWAAAAHgOQSwAAAA8hyAWSMDOnTtNVFSUmTFjRqSLgougbTh06NBk/51OnTqZ7Nmzm0vdzJkzTbly5UzGjBlN7ty57bR69erZV0qL1O9KiRIl7DZzLV++3O4r+n9aF7xdEnMuVJ2qboHkRBCLNE8nZJ2YQ70GDBiQLL/57LPPmgULFiTLsr3snXfeMS+++GLEfv/PP/+0ge7atWtNavbrr7/aIKN06dJmypQp5vXXX490kQAfzo8IV4aw5wRSueHDh5uSJUsGTLv66qtN8eLFzYkTJ2zGKilP0q1btzZ33HFHki0ztQSxv/zyi+nVq1fEgthhw4bZDFLVqlVNaqUs4/nz581LL71kypQp45u+ePFik9bddNNN9njPlClTpItyyUmOc2EonB8RLoJY4H+aNm1qatSoEfKzzJkzJ/j9Y8eOmWzZsiVDybzr+PHjJmvWrJEuBoLs37/f/t9tRuAicDMmXbp0YR3vaZGeTlE3uJTQnABIQKh2YG67x23btplbb73V5MiRw3To0MF+tmXLFtOqVStTqFAhe8K//PLLTbt27czhw4ft51qWAt4333zT12zBv01eKLt27TLNmze3QXKBAgXM448/bj7//POQbfd++OEH06RJE5MrVy4bQNatW9esWLEiYB49Mtd3t27dan9bwYzmv//++23gGeztt9821atXN1myZDF58+a167Nnz56AedR2Tpnrn376yWaz9NtPPvmk/ezDDz80zZo1M0WKFDHR0dH2MfYzzzxjzp07F/D9Tz75xK6rWy/+bepOnTplhgwZYjOHWkaxYsVMv3797HR/eq/6yZ8/v90uqre9e/eahKger732Wvtv1YNbBv/t/u677/rqIV++fOaee+4xf/zxR4LLVvMElUfrePToUTtN33vggQdMwYIF7fpUrFjRTJs2LVaZVIZ58+aZkSNH2n1J+1SDBg3stvOn7aZmAgcOHIi3LKpT1aOoTP5thf3bQP79998mQ4YMNjMd7LfffrPfe+WVV3zT/vvvP5tB13bR+mg7jRkzxmZ8L0S423v69Onm5ptvtseF5qtQoYJ59dVXYy3PcRwzYsQIW4faN+vXr282btwYa75QbWLdfXvTpk32e/p+0aJFzXPPPXdRx2oo2i86d+7sO1b0dKhHjx7m9OnT9vN///3XPPHEE6ZSpUr2HJQzZ057A75u3boL3ndETUp0XGrfrlmzpvnmm29izRNXm1g9+lf9aPn6/wcffBBy3caNG2fq1KljLrvsMvs7Opbee++9gHkSOj+Gc9zIhAkT7GfaVnny5LEJCj3pQepCJhb4HwWZwQGAApW4nD171jRu3NjccMMN9uSsk6UuNJqmC+2jjz5qA1mddD/++GN7kVegqA41Xbp0sReKbt262WXp4hEXndB1kd63b5957LHH7DJ1Mv7yyy9jzfvFF1/YC5ouDgoAlFVyL/K6KOk3/bVp08ZeJEeNGmV+/vlnM3XqVHvhVfDh0gVw0KBBdl6V+59//rEXCAWqa9asCcjmHTx40P6+glwFeLrQiC56uuD27t3b/l/lHDx4sImJiTFjx4618zz11FN2GyjgfOGFF+w0t4OUAiEFBt9++62ts/Lly5sNGzbY+X7//feA9nMqo4Lu9u3b2wumfksBdEK0TDUpUbn0GzfeeKOdrmW466DgVoGu6ktBnh7H6wYhuB78/fjjj3af0EVUwbwu3vpu7dq17QX6kUcescHkp59+aoMX1Ulwc4rRo0fbbangRXWk4Ek3Tbphca1atcoGWNru8XVgU5vjt956ywYaCvZUx5UrV441n7adboAUBLlBr2vu3Lkmffr05q677vIF0JpX+3r37t3NFVdcYb777jszcOBAu98mtp1zYra31kHBiuZX0P3RRx+Zhx56yC7j4Ycf9s2n7aogVjedeml/v+WWW3zBYUIOHTpkbw5btmxpjwUFX/3797fBpPb5xB6rcTVn0TGqc4XWWx3vVKf6LdWxMuXbt2+366+617GrfWny5Mm2/hVkK/hN7L7zxhtv2O2mfV37nn5D9akbVt08xEdNUHTTrpsHHRc6B+g4UdAcTMeLlqvfV73PmTPHrofOj+4xGt/5MdzjRu28e/bsaZskaDucPHnSrF+/3q6zzgtIRRwgjZs+fbqjQyHUS3bs2GH/rflcHTt2tNMGDBgQsKw1a9bY6e+++268v5ktWza7jHCMHz/eLnPBggW+aSdOnHDKlStnp3/55Zd22vnz550rr7zSady4sf236/jx407JkiWdRo0a+aYNGTLEfveBBx4I+K0777zTueyyy3zvd+7c6aRPn94ZOXJkwHwbNmxwMmTIEDC9bt26dpmvvfZarHVQGYJ1797dyZo1q3Py5EnftGbNmjnFixePNe/MmTOddOnSOd98803AdP2WfnPFihX2/dq1a+37hx56KGC+9u3b2+la7/j8+OOPsba1nD592ilQoIBz9dVX27p3ffzxx3b+wYMH+6Zpu2r7yrfffuvkzJnTrpf/enbu3NkpXLiwc+DAgYDfadeunZMrVy5ffWnbavnly5d3Tp065ZvvpZdestO1HVzuvAmto//2/+effwKmaxvq5Zo8eXKs35EKFSo4N998s+/9M888Y9f5999/D5hPx4f2n927d8dbnuDfDXd7x7Vv6RgoVaqU7/3+/fudTJky2e3gf2w8+eSTdnn+x6Jbj+5x5ZZP09566y3fNG2PQoUKOa1atUr0sRqX++67z6639sNgbrm1H507dy7gM52joqOjneHDh8daj4T2HXffrlq1asB8r7/+up3Pf7uEOhfqe9qX//vvP9+0xYsX2/mCj+XgbaXf1jHlvy/Fd34M97hp0aKFU7FixVjfR+pDcwLgfyZOnGiWLFkS8EqIHvP5U6ZV9Pgw1GP5C/HZZ5/ZR5fKYLj02K5r166xHlmrKYMyDcqGKKusl7JDeoT49ddfx3q0++CDDwa8V/ZR31VWQ+bPn2+/o8yTuzy9lGG68sorY2WY9HhPWZhgyj66jhw5Ypeh33IfgSdEj/GVjVNmyr8cynqJW45FixbZ/ysL4+9iO4qtXr3atiNVhs+/TaCyRyqTmkEEU5mUgVXdqx5VN+5j7ffff9/cfvvt9t/+66P5lS1TltCf6tS/vaqbJVbGzP+Rt5aXlMOIKeuo7KYyry51vFPGr23btgHbR2XSY1v/9WnYsKFtMqJ9LzHC3d7B+5b7NEVZSdWN24Rn6dKlNvOnpyPK4l3IfqGMtZ4uuLQ9lC303wbhHquh6DhThlX7Rai2+W65tR8psyqqWx2vKttVV10Va78JZ99x922dC/zn0yN893wWF2Wcdd7p2LFjwLyNGjWymdlg/ttKmW1tH5UnVLmDJea40VMRPdHRUxCkbjQnAP5HF6S4OnaFoot78CMzPd7TI/Pnn3/ezJo1y56gdUHTxS+hC0Jc1MZOj9P8L77i36tcFMCKLihx0YlegYZLj339uZ/pAqO2dlqmLhgKWEMJ7qWsC3iozkFqe/j000/bR/tugOxfpoSoHJs3b7aPD+PrqKS60gU+uHmGLvAXQ8uNazkKtPTY258eXyrAVbMOPY7XvuJScww9LlYbxLiGtnLXJ5ztlJzUnEZBuNZBbZhFAa3WRwGu//bR49qEtk+4wt3eouYcau6wcuXKWDeO2rd03LnbL3g/1vL9j4f46FgPPgb1Xa13Yo/VULRf6NhQm9L4uKNKTJo0yezYsSOgXbnamgZLaN+Jq250bJcqVSressT1XQkVVKvZgJp0KPD1b9scXF+hJOa4UTMP3bjonK66V7MR3dxff/31Cf4OvIUgFrhA/hkRf+PHj7dZDLV/VHsxZQXVVuz7778P2U4sqbhZVrUxjWt4qOBB+NWuMRQFru4ydYFRu7NQ8wYvzz/T4tKFR5kxBcVqc6qLvLJTusDpYhNOxx/No7aHujkIJaF2e5HYN9TuUvuAsnO33Xab7zN3fXVjE9cNR3Ab1YS2U3JS+2Zl8xR4aL9SQKvA1r+9uNZJ2Td1vAqlbNmyifrNcLe3OlaqLLqR0LyarpsoZeTVfvZCO5WFEsltEDz8lNqoq3OTbizUblXnIWWVQ63vpVJutcnXDb3a0isAL1y4sA2U1WY/nA5XiTlulMVX50MFzTr+lMHVb6pddKiOivAuglggGegCrJeyj+rgogzAa6+9ZrMQ4WYe/Mdm1ONbXXT8vxfcw9jNPipY1GPcpKBl6neVYU5sIOLfS1qPPPVIXRcwl7JIweKqF5VDva8VsMRXd6orXewU3PhnTXVBC0dcy9Zy3eW4j7T9l+1+7r8cZeJbtGhhO67oJsDt9e+OmqAMWlJtp+SksTrV6cdtUqCOVeqwFbx9NOpCUu534WxvdeJSRm/hwoUBGcfgZi7u9lGG1z+7qOxeUmazwz1WQ9F+oWNXzTXio05e6sCnzljBN4vxdUSNr8xu3fjv22fOnLHHaJUqVcL6brDgY06BpG5e1dTKbVojCmKDhdrmiT1uNDqEmrzopaYkenKgTqradxkmLPWgTSyQhPQ4UKMW+FMwq0yJ/+MznWB10QmH2nuph7Iu1P6Pq9UD158eXevir5ES3GGc/OmCnVg68SuTo+xFcOZG7xWcJsTNBPl/XxcVZUaCqV5CNS9Qm1zVQfA6iwZfV7tfcXuJv/zyywHzhNs73h3nN3jbqJmJRm3QjYj/dlRwqsfeoUY/UEZQgbtGM1A7Po0e4NaHenProh4qYLmQ7ZSYIbYSS+0LtQ8qA6ve5Fqv4EHotX30OF8BSjDVZfAxkZBwt3eofUv7T3BgpKBHWT+NquE/b1L/dbhwj9VQdI5QvSowVzvVYG65tc7Bx6LaEIcz1Fso2rcVIGrf9h+pQaNxJHSOUjZV2XkNh+V/3Ko/gYJ5fyq3glP/5g8asivUX+YKdX5MzHETfF7SPqs2uqo3BedIPcjEAklIbT419Iuyb8pc6uKtIWPcE7B/wKk2W3oEqiFxlOmsVatWyGUqC6bxOO+++247XIwuHMryudkEN2uhi6CGyFIgpyGH9AhYbVR1cVNmSlkeXSATQ0GxssfKXuiCo4ussiHK0GiIJg2Bo6F74qNhe9QOT48A1bRC5VWdhHqcqXpRxk/tihX8qbmCAsB7773XBlHqfKJ1UWZbF0MFbZqu4EkXY11QVU8KkHVR1W8vW7YsrEyYu74K2nRB13rqYqrtou2jYcdUp2oaod9wh9jSuKsaCzQUNa/QI01luLRdvvrqK9vmUcMeaT20bHX60QVW43+qiYX2C/07scIdYutCKJulx7iqVwVqwcOJ9e3b1wZuajahpjTajgo0NSyWMofadxKTJQx3e6utowIU7SM6TnTzpoBRNxzqdORSkKb9VM16VEY19dCwaLoJuZDsZVzCPVbjayqgJkjax9yhxbQeClLV7lr1rvKrWY72Re3fqmP9RkLtV+Oi4F7HuMqu/VTbWse3bgTCWabqVDdxGmpQTRy077pjtPrfTGsene80TJnap6r9qjrTqs2qf7vi+M6P4R432i/U+VT7jYaK042mtovKoOMaqUikh0cALpUhtkINaxPfEFvuMEr+tm/fboetKl26tJM5c2Ynb968Tv369Z2lS5cGzPfrr786N910k5MlS5ZYQ/yEouVqeCDNnz9/fqdPnz7O+++/b7/7/fffxxrmq2XLlnaoLA27o2Fu2rRp4yxbtizBIZbcutA6+9Nv3XDDDXad9dKQQQ8//LDz22+/+ebRUDxxDWujIZFq165ty1+kSBGnX79+zueffx5r2KGjR4/a4bBy584da4geDcczZswY+xtarzx58jjVq1d3hg0b5hw+fDhgSKOePXva9VdZb7/9dmfPnj1hDz/14Ycf2iGkNIRY8HafO3euU61aNfv72rYdOnRw9u7dG/D9UPuGhgTSMjUk05YtW+y0v//+29ZhsWLFnIwZM9rPGjRoYIc2Ch4mKXjItlD7ZHIMseWKiYnx7atvv/12yGUeOXLEGThwoFOmTBk7nFW+fPmcOnXqOOPGjbPbLj6hfjfc7b1w4UKncuXK9ngrUaKE/c60adNi7ccalkrf1RBNWpd69eo5v/zyi93HwhliK9S+re8FDyOVmGM1lF27dtmhtvRdrbeGCtN+4g5/pSG2tEx3Pa6//npn5cqVseowMfuOTJo0yQ7Fp9+sUaOG8/XXX8daZlzf1fppKC99V/v5/PnzQ9bNG2+8YYcB1Hw6h2g57r4Y7vkxnONGQ8Pp++45UOfjvn37Buw3SB2i9J9IB9IAEk+PQpUB1FAyyrgCuDRxrALJgyAW8AC1A/Tv+a92dtWqVbOPWNXRBsClgWMVSDm0iQU8QB2s1PtabT7V1lN/VlXtA9UWDsClg2MVSDkEsYAHqDONOm3pQqiMjjo0qKe4/19NAhB5HKtAyqE5AQAAADyHcWIBAADgOQSxAAAA8BzaxCYx/cnLP//80w6onJg/LQoAAABj/xjOkSNH7B+70B/yiQtBbBJTAFusWLFIFwMAAMDT9uzZYy6//PI4PyeITWLun7RTxevPfAIAACB8MTExNiGY0J8JJohNYm4TAgWwBLEAAAAXJqFmmXTsAgAAgOcQxAIAAMBzCGIBAADgOQSxAAAA8ByCWAAAAHgOQSwAAAA8hyAWAAAAnsM4scnk6iGfm3TRWSNdDAAAgIu2c3Qzc6khEwsAAADPIYgFAACA5xDEAgAAwHMIYgEAAOA5BLEAAADwHIJYAAAAeA5BLAAAADyHIBYAAACeQxALAAAAz0m1QeyMGTNM7ty5I10MAAAApPUgtlOnTvYVFRUV70sBbFLR8nbu3JlkywMAAMDFy2A8plixYmbfvn2+9+PGjTOfffaZWbp0qW9arly5zNy5cyNUQgAAACQ3T2ViJX369KZQoUK+V/bs2U2GDBkCpmXJksU3/+eff27Kly9v52vSpElAACxTp061n2fOnNmUK1fOTJo0KQJrBQAAgFSdiU2M48eP20ztzJkzTbp06cw999xjnnjiCTNr1iz7uf4/ePBg88orr5hq1aqZNWvWmK5du5ps2bKZjh07hvUbp06dsi9XTExMsq0PAAAAPBjEJrat65kzZ8xrr71mSpcubd8/8sgjZvjw4b7PhwwZYsaPH29atmxp35csWdJs2rTJTJ482RfEOo4T72+MGjXKDBs27ALWBgAAAGmmOUFiZM2a1RfASuHChc3+/fvtv48dO2a2bdtmOnfubJsauK8RI0bY6eEaOHCgOXz4sO+1Z8+eZFkXAAAAeDQTm1gZM2aMNdKAm1k9evSo/f+UKVNMrVq1YrW7DVd0dLR9AQAAIOWk6iA2PgULFjRFihQx27dvNx06dIh0cQAAAJAIaTaIFbVl7dmzpx2SSyMXqIPW6tWrzaFDh0zv3r0jXTwAAADEIU0HsV26dLHtZseOHWv69u1rRyWoVKmS6dWrV6SLBgAAgHhEOQl1v0eiaIgtZXaL9Zpn0kVnjXRxAAAALtrO0c1MSsdS6jCfM2fOtDk6AQAAAFInglgAAAB4DkEsAAAAPIcgFgAAAJ5DEAsAAADPIYgFAACA5xDEAgAAwHMIYgEAAOA5BLEAAADwnDT9Z2eT0y/DGsf7VyYAAABw4cjEAgAAwHMIYgEAAOA5BLEAAADwHIJYAAAAeA5BLAAAADyHIBYAAACewxBbyeTqIZ+bdNFZI10MIOJ2jm4W6SIAAFIhMrEAAADwHIJYAAAAeA5BLAAAADyHIBYAAACeQxALAAAAzyGIBQAAgOcQxAIAAMBzCGIBAADgOQSxAAAA8JxUG8QOHTrUVK1aNdLFAAAAQGoKYjt16mQDTdfGjRtNmzZtTP78+U10dLQpW7asGTx4sDl+/LiJpOXLl5sSJUpEtAwAAAC4BDOx33//valVq5Y5ffq0+eSTT8zvv/9uRo4caWbMmGEaNWpkp8flzJkzyVau5Fw2AAAAPBzEOo5jOnfubMqXL2/mz59vatasaYoXL27uuusu89FHH5mVK1eaF154wTd/VFSUefXVV03z5s1NtmzZbLAro0ePNgULFjQ5cuSwyzt58mSs35o6dar9ncyZM5ty5cqZSZMm+T7buXOnXfbcuXNN3bp17TyzZs1KoVoAAACAp4LYtWvXmk2bNpnevXubdOkCi1OlShXTsGFDM3v27IDpaoZw5513mg0bNpgHHnjAzJs3z0579tlnzerVq03hwoUDAlRRQKrmCQp6N2/ebOcdNGiQefPNNwPmGzBggHnsscfsPI0bN07GNQcAAMCFymAiRE0FRJlPUYY0FE3/9ttvA6a1b9/e3H///b737dq1s9lXvWTEiBFm6dKlAdnYIUOGmPHjx5uWLVva9yVLlrTB8+TJk03Hjh198/Xq1cs3jyggVpY2LqdOnbIvV0xMTCJqAQAAAJ7MxPo3KwhXjRo1At4ra6o2tf6uu+4637+PHTtmtm3bZoPc7Nmz+14KdjU9vmUnZNSoUSZXrly+V7FixRL1fQAAAHgoE+vSKARuIFqtWrVYn2u6O49LbWET4+jRo/b/U6ZMiRXspk+f/qKWPXDgQNsUwj8TSyALAACQyjOxGstVnazUeev8+fMBn61bt842C7j77rvjXYaaHPzwww+xRjxwqcNXkSJFzPbt202ZMmUCXmpWcDE0HFjOnDkDXgAAAEjlmViNCPDGG2/YobRatWplM5uFChWyQWmfPn1sswC1U42POmJp3Fk1Bbj++uttJy6NO1uqVCnfPMOGDTM9e/a0j/ybNGli27GqE9ihQ4cCMqkAAAC49EU8Eyt16tSxmVM92m/atKnNkCqYVYerJUuW2GxnfNq2bWtHGujXr5+pXr262bVrl+nRo0fAPF26dLFDbE2fPt1UqlTJDqOlzmUXm4kFAABAyotyEtOjCglSm1jbwavXPJMuOmukiwNE3M7RzSJdBACAB2Opw4cPx9tM85LIxAIAAACJQRALAAAAzyGIBQAAgOcQxAIAAMBzCGIBAADgOQSxAAAA8ByCWAAAAHgOQSwAAAA8hyAWAAAAnpMh0gVIrX4Z1jjevzIBAACAC0cmFgAAAJ5DEAsAAADPIYgFAACA5xDEAgAAwHMIYgEAAOA5BLEAAADwHIJYAAAAeA7jxCaTq4d8btJFZ410MYCLtnN0s0gXAQCAWMjEAgAAwHMIYgEAAOA5BLEAAADwHIJYAAAAeA5BLAAAADyHIBYAAACeQxALAAAAzyGIBQAAgOcQxAIAAMBzCGIBAADgOWkuiO3UqZMZOnRowLTZs2eb9OnTm4cffjjW/MuXLzclSpRIwRICAAAgIWkuiA3ljTfeMP369bPB7MmTJyNdHAAAACQgzQexO3bsMN99950ZMGCAKVu2rJk/f36kiwQAAIAEpPkgdvr06aZZs2YmV65c5p577rFZWQAAAFzaohzHcUwadf78edvedcKECaZFixbmwIEDpmjRoubXX381JUuWDGsZp06dsi9XTEyMKVasmCnWa55JF501GUsPpIydo5tFuggAgDQkJibGJhcPHz5scubMGed8aToTu2TJEnPs2DFz66232vf58uUzjRo1MtOmTQt7GaNGjbIV7b4UwAIAACB5pekgVk0H/v33X5MlSxaTIUMG+1q0aJF58803bZY2HAMHDrR3Cu5rz549yV5uAACAtC6DSaMOHjxoPvzwQzNnzhxTsWJF3/Rz586ZG264wSxevNg0adIkweVER0fbFwAAAFJOmg1iZ86caS677DLTpk0bExUVFfCZmhcoSxtOEAsAAICUl2abE6jd65133hkrgJVWrVqZhQsX2o5eAAAAuPSk2Uzs+vXr4/xM2Vm9AAAAcGlKs5lYAAAAeBdBLAAAADyHIBYAAACeQxALAAAAzyGIBQAAgOcQxAIAAMBzCGIBAADgOQSxAAAA8ByCWAAAAHhOmv2LXcntl2GNTc6cOSNdDAAAgFSJTCwAAAA8hyAWAAAAnkMQCwAAAM8hiAUAAIDnEMQCAADAcwhiAQAA4DkEsQAAAPAcxolNJlcP+dyki84a6WIggnaObhbpIgAAkGqRiQUAAIDnEMQCAADAcwhiAQAA4DkEsQAAAPAcglgAAAB4DkEsAAAAPIcgFgAAAJ5DEAsAAADPIYgFAACA51xSQezQoUNN1apVI10MAAAApPUgtlOnTjY4lXr16pmoqCj7ypw5s6lQoYKZNGmSuZSpzDNmzIh0MQAAABDJTGzXrl3Nvn37zKZNm0ybNm3Mww8/bGbPnp3SxQAAAICHpXgQmzVrVlOoUCFTqlQpm6G98sorzcKFC+Ocf+rUqaZ8+fI2c1uuXLlYmdv+/fubsmXL2uVqmYMGDTJnzpzxfb5u3TpTv359kyNHDpMzZ05TvXp1s3r1at/n3377rbnxxhtNlixZTLFixUzPnj3NsWPHkmntAQAAkBQymAhT8Hj69OmQn82aNcsMHjzYvPLKK6ZatWpmzZo1NpObLVs207FjRzuPglM97i9SpIjZsGGD/VzT+vXrZz/v0KGD/e6rr75q0qdPb9auXWsyZsxoP9u2bZtp0qSJGTFihJk2bZr5559/zCOPPGJf06dPD6v8p06dsi9XTExMEtQKAAAAIhrExtWe9Ny5c7YZwfr16023bt1CzjNkyBAzfvx407JlS/u+ZMmSthnC5MmTfUHs008/7Zu/RIkS5oknnjBz5szxBbG7d+82ffv2tVlcUebXNWrUKBvk9urVy/fZyy+/bOrWrWuDXmV/ly9fHu/6aRnDhg1LZK0AAADgYkQ5juOYFOwk9d1335lMmTLZ7Ksyoz169DDjxo0z6dKls80LFixYYLOleqSfPXt2m6nVZ66zZ8+aXLlymb///tu+nzt3rg08lVU9evSo/VzNBvbv328/1zJHjhxpA9OGDRuau+66y5QuXdp+du2119og2s3Miqrj+PHjNlhWM4YLycSqWUKxXvNMuuisSVp/8Jado5tFuggAAHiOYinFeocPH7Yx3SXTnECZz6eeesoGp4ULFw4IUP0pIJUpU6aYWrVqBXym4FdWrlxpl6dMaOPGje0KKwur7K1LQWz79u3NJ598Yj799FOb3dU8d955p/2N7t2723awwa644oqw1ic6Otq+AAAAkHJSPIhVoFmmTJkE5ytYsKBt57p9+3YbqIairG7x4sVtUOzatWtXrPnU8Uuvxx9/3Nx99922vauC2GuuucZmXMMpDwAAAC4dEe/YFR9lWJUlVeCrDlh6bK+RBQ4dOmR69+5t27Cqzasyq2oaoGzrBx984Pv+iRMnbHvY1q1b2/a0e/fuNT/++KNp1aqVb2SD2rVr245cXbp0sR3GFNQuWbLEdiYDAADApemS+otdwRRYaogtZU4rVapk27Wqo5gCUmnevLnNrioI1V/6UmZWQ2z5Nzs4ePCgue+++2wmVuPSNm3a1NcRq3Llyuarr74yv//+ux1mS6MYaDQEZYABAABw6UrRjl1pqTEyHbtAxy4AAJKvY9clnYkFAAAAQiGIBQAAgOcQxAIAAMBzCGIBAADgOQSxAAAA8ByCWAAAAHgOQSwAAAA8hyAWAAAAnkMQCwAAAM/JEOkCpFa/DGsc71+ZAAAAwIUjEwsAAADPIYgFAACA5xDEAgAAwHMIYgEAAOA5BLEAAADwHIJYAAAAeA5DbCWTq4d8btJFZ410MZCEdo5uFukiAACA/yETCwAAAM8hiAUAAIDnEMQCAADAcwhiAQAA4DkEsQAAAPAcglgAAAB4DkEsAAAAPIcgFgAAAJ5DEAsAAADPuSSC2E6dOpk77rjD975evXqmV69eES0TAAAAUkkQq2Bz6NChvkAzKirKvqKjo03RokXN7bffbubPn3/RhdIynnnmGXMp0Prt3Lkz0sUAAABAUmViu3btavbt22e2bdtm3n//fVOhQgXTrl07061bt4tZrMmbN6/JkSOHSU5nzpxJ1uUDAADgEg1is2bNagoVKmQuv/xyU7t2bTNmzBgzefJkM2XKFLN06VLffHv27DFt2rQxuXPntgFqixYt4s1u+jcnePLJJ02tWrVizVOlShUzfPhw3/upU6ea8uXLm8yZM5ty5cqZSZMm+T7TbymjOnfuXFO3bl07z+uvv25y5sxp3nvvvYDlLliwwGTLls0cOXLkYqoGAAAAXmoT27FjR5MnTx5fswJlPBs3bmwzq998841ZsWKFyZ49u2nSpIk5ffp0gsvr0KGDWbVqlc32ujZu3GjWr19v2rdvb9/PmjXLDB482IwcOdJs3rzZPPvss2bQoEHmzTffDFjWgAEDzGOPPWbnadmypc0aT58+PWAevW/dunWyZ4IBAABw4TIkZuYZM2YkOE+6dOlM2bJlfZlWZT/Pnz9vM6XKhrqBorKyy5cvN7fccku8y6tYsaLNur7zzjs2MHWDVmVny5QpY98PGTLEjB8/3gamUrJkSbNp0yabFVZQ7VJ2151HunTpYurUqWObRBQuXNjs37/fLFq0KCCL7DhOvOU7deqUfbliYmISrCMAAABcgqMTKPBzA9Z169aZrVu32symMrB6qUnByZMnA7KrCWVjFcS6y549e7adJseOHbPL6dy5s2/5eo0YMSLW8mvUqBHwvmbNmjZIdjO2b7/9tilevLi56aabwl7XUaNGmVy5cvlexYoVC/u7AAAASIFMbDjOnTtntmzZYq699lr7/ujRo6Z69eo2exosf/78YS3z7rvvNv379zc///yzOXHihG1j27ZtW9/yRe1wg9vOpk+fPuC92roGUzZ24sSJtqmBMsT333+/LwAPx8CBA03v3r0DMrEEsgAAAB4LYpXVPHTokGnVqpV9f80119gmBQUKFLAdqS6EOo6pQ5YCYQWxjRo1ssuTggULmiJFipjt27f7srOJcc8995h+/fqZl19+2TZB8G9+EA4NL6YXAAAAPBLEHj9+3Pz111/m7NmzZu/eveaDDz4wL7zwgunRo4epX7++nUeB5dixY+2IBBpNQAHprl27bMcvBY96Hw4tR21f1RlMv+Fv2LBhpmfPnvZxvjqMqY3q6tWrbTDtnyUNRZ3Q1E62b9++tn1uuOUBAACAR9vE6hG+OkSVLl3aBoLKZCrr6j+8lYbh+vrrr80VV1xh59EwWGq/qjaxicnMasSAgwcP2sDZ/697uU0C1HFMzQEqVapks7bqhKYOXuFQeRQcP/DAA4lYewAAAERKlJNQ9/s0YObMmebxxx83f/75p8mUKdNFLUttYm0Hr17zTLrorElWRkTeztHNIl0EAABSvZj/xVKHDx+ON+GZ5G1ivURZXQ2vNXr0aNO9e/eLDmABAADg4SG2vOK5556zf91Lf3VMowwAAADAG9J0EDt06FD7F8WWLVtmx5YFAACAN6TpIBYAAADeRBALAAAAzyGIBQAAgOcQxAIAAMBzCGIBAADgOQSxAAAA8Jw0/ccOktMvwxon6s/qAgAAIHxkYgEAAOA5BLEAAADwHIJYAAAAeA5BLAAAADyHIBYAAACeQxALAAAAzyGIBQAAgOcQxAIAAMBz+GMHScxxHPv/mJiYSBcFAADAc9wYyo2p4kIQm8QOHjxo/1+sWLFIFwUAAMCzjhw5YnLlyhXn5wSxSSxv3rz2/7t374634pE8d266edizZw9/8jeFUfeRQ91HDnUfOdR95KRE3SsDqwC2SJEi8c5HEJvE0qX7/2bGCmA5sCJD9U7dRwZ1HznUfeRQ95FD3afeug8nEUjHLgAAAHgOQSwAAAA8hyA2iUVHR5shQ4bY/yNlUfeRQ91HDnUfOdR95FD3kXMp1X2Uk9D4BQAAAMAlhkwsAAAAPIcgFgAAAJ5DEAsAAADPIYgNw8SJE02JEiVM5syZTa1atcyqVavinf/dd9815cqVs/NXqlTJLFq0KOBzNUMePHiwKVy4sMmSJYtp2LCh2bJlSzKvhTcldd136tTJREVFBbyaNGmSzGuR+ut+48aNplWrVnZ+1emLL7540ctMq5K63ocOHRprn9cxgour+ylTppgbb7zR5MmTx750Hg+en3N95Oqec33y1P38+fNNjRo1TO7cuU22bNlM1apVzcyZMyO336tjF+I2Z84cJ1OmTM60adOcjRs3Ol27dnVy587t/P333yHnX7FihZM+fXrnueeeczZt2uQ8/fTTTsaMGZ0NGzb45hk9erSTK1cuZ8GCBc66deuc5s2bOyVLlnROnDiRgmuWNuu+Y8eOTpMmTZx9+/b5Xv/++28KrlXqrPtVq1Y5TzzxhDN79mynUKFCzgsvvHDRy0yLkqPehwwZ4lSsWDFgn//nn39SYG1Sd923b9/emThxorNmzRpn8+bNTqdOnex5fe/evb55ONdHru451ydP3X/55ZfO/Pnz7TV269atzosvvmivu5999llE9nuC2ATUrFnTefjhh33vz5075xQpUsQZNWpUyPnbtGnjNGvWLGBarVq1nO7du9t/nz9/3l5sxo4d6/v8v//+c6Kjo+2FCMlX9+6JrUWLFslY6rRZ9/6KFy8eMpi6mGWmFclR7wpiq1SpkuRlTW0udv88e/askyNHDufNN9+07znXR67uhXN9eJLivFytWjWbNIrEfk9zgnicPn3a/PTTTzYV7v9nZfV+5cqVIb+j6f7zS+PGjX3z79ixw/z1118B8+hPqymFH9cy06LkqHvX8uXLTYECBcxVV11levToYQ4ePJhMa5F26j4Sy0xtkrOO9ChPf4O8VKlSpkOHDmb37t1JUOLUIynq/vjx4+bMmTMmb9689j3n+sjVvYtzffLWvRKhy5YtM7/99pu56aabIrLfE8TG48CBA+bcuXOmYMGCAdP1XhspFE2Pb373/4lZZlqUHHUvahP11ltv2QNvzJgx5quvvjJNmza1v4ULr/tILDO1Sa460sVjxowZ5rPPPjOvvvqqvcioPeGRI0eSoNSpQ1LUff/+/e2Ngnvx5lwfuboXzvXJV/eHDx822bNnN5kyZTLNmjUzEyZMMI0aNYrIfp8hyZcIXMLatWvn+7c6flWuXNmULl3a3rE3aNAgomUDkoMu3C7t7wpqixcvbubNm2c6d+4c0bKlFqNHjzZz5syx5xF1jkHk655zffLJkSOHWbt2rTl69Ki9Sejdu7d9ylOvXj2T0sjExiNfvnwmffr05u+//w6YrveFChUK+R1Nj29+9/+JWWZalBx1H4oOPP3W1q1bk6jkabPuI7HM1Cal6ki9isuWLcs+n0R1P27cOBtILV682AZKLs71kav7UDjXJ13dq8lBmTJl7MgEffr0Ma1btzajRo2KyH5PEBsPpcqrV69u7zRc58+ft++vu+66kN/RdP/5ZcmSJb75S5YsaTek/zwxMTHmhx9+iHOZaVFy1H0oe/fute2kNBQILrzuI7HM1Cal6kjZk23btrHPJ0HdP/fcc+aZZ56xTTU07JA/zvWRq/tQONcn3zlH3zl16lRk9vsk7yqWCoefUK+6GTNm2CElunXrZoef+Ouvv+zn9957rzNgwICAYZ4yZMjgjBs3zg79oZ7BoYbY0jI+/PBDZ/369bYHJcOuJH/dHzlyxA5HtHLlSmfHjh3O0qVLnWuuuca58sornZMnT0ZsPVND3Z86dcoOd6NX4cKFbT3r31u2bAl7mUieeu/Tp4+zfPlyu8/rGGnYsKGTL18+Z//+/RFZx9RS9zqPa2ii9957L2AYJ51n/OfhXJ/ydc+5Pvnq/tlnn3UWL17sbNu2zc6v662uu1OmTInIfk8QG4YJEyY4V1xxhT1oNBzF999/7/usbt26digPf/PmzXPKli1r59f4jJ988knA5xqCYtCgQU7BggXtztOgQQPnt99+S7H1Sat1f/z4ceeWW25x8ufPb4NbDUmkMfEIoi6+7nWh0D1x8EvzhbtMJE+9t23b1ga4Wl7RokXte43viIure50/QtW9bp5dnOsjU/ec65Ov7p966imnTJkyTubMmZ08efI41113nQ2E/aXkfh+l/yR9fhcAAABIPrSJBQAAgOcQxAIAAMBzCGIBAADgOQSxAAAA8ByCWAAAAHgOQSwAAAA8hyAWAAAAnkMQCwAAAM8hiAUQtqFDh5qqVaua1GLGjBkmd+7cF72cEiVKmBdffDHeeaKiosyCBQvsv3fu3Gnfr1271r5fvny5ff/ff/+Z5DBo0CDTrVs333v9jRu9z5s3b0A5EhLfOlyqwtk2CUnu7ZOcLnRbp5QDBw6YAgUKmL1790a6KPAgglggmXXq1MlePEaPHh0wXcGApkfigp3aglEv2Ldvn2natGnIz+rUqWM/z5UrV5IG1/LXX3+Zl156yTz11FO+aZ999pn9jY8//tj+7tVXX50kv4VLT6ht7X8zEmn58uUz9913nxkyZEikiwIPIogFUkDmzJnNmDFjzKFDhyJdlDTh3Llz5vz58+ZSUqhQIRMdHR3ys0yZMtnPk/qmRqZOnWqD5OLFi/umbdu2zRQuXNhO1+9myJAhyX8XgU6fPh2R3/XCtr7//vvNrFmzzL///hvposBjCGKBFNCwYUN7ARk1alS883377bfmxhtvNFmyZDHFihUzPXv2NMeOHbOfvfXWWyZ79uxmy5YtvvkfeughU65cOXP8+HFTr149s2vXLvP444/bYCiugEhZmWHDhpl169b55tM02b17t2nRooX9nZw5c5o2bdqYv//+O94LZKlSpcwjjzxiH1ueOnXKPPHEE6Zo0aImW7ZsplatWvZRrP9vK8P4+eefm/Lly9vfadKkic0QJfQo95NPPjGVK1e2NwS1a9c2v/zyS6zlLly40FSoUMEGi1oX3TQoy5MnTx6TNWtWmwn1rz+XslJXXnmlXXbjxo3Nnj17AtZRdVKwYEFb3muvvdYsXbo01jKOHDli7r77brveWv+JEycGfB5f9sv/cbX+rYv64cOHfdtHmfPhw4eHzJgqo67mAnGZM2eOuf322wOeDDz66KO2frRsZe/jyuJr2frtC/Hkk0/a7R+sSpUqdl1ENxr69+WXX263mX5PmUN/esysetXjcNVtjRo1zA8//JBk2yZUswhtB03z33f9HTx40C5Py9J+ValSJTN79uyAeXQ86rjo1auXzTZqv3rggQfMbbfdFjDfmTNn7OP0N954I+Rv6ZjW9tM+rPJXrFjRLFq0yPf5V199ZWrWrGnrT8HqgAEDzNmzZ+Pc1u72vvPOOwO2v/t0Ztq0aeaKK66w9anzi24In3vuOXv+UjlHjhwZUL7nn3/err/KpnOWvnP06FHf51pnHbc6N7jBfLVq1exx6dI6FSlSxHzwwQch6wCIkwMgWXXs2NFp0aKFM3/+fCdz5szOnj177PQPPvjA8T8Et27d6mTLls154YUXnN9//91ZsWKFU61aNadTp06+ee666y7n2muvdc6cOeN8/PHHTsaMGZ3Vq1fbzw4ePOhcfvnlzvDhw519+/bZVyjHjx93+vTp41SsWNE3n6adO3fOqVq1qnPDDTfYZX7//fdO9erVnbp16/q+O2TIEKdKlSr23+vWrXMKFSrkPPXUU77Pu3Tp4tSpU8f5+uuv7fqMHTvWiY6Otusj06dPt2Vu2LCh8+OPPzo//fSTU758ead9+/Zx1t+XX35p60nzLV682Fm/fr1z2223OSVKlHBOnz4dsFz9turt119/dY4dO+Y0b97cfk/lWbt2rdO4cWOnTJkysb5Xo0YN57vvvrPrXbNmTbscl7732muvORs2bLDr8fTTT9vtuGvXLt88xYsXd3LkyOGMGjXK+e2335yXX37ZSZ8+vS2vS+ugbS47duyw79esWROwjocOHXJOnTrlvPjii07OnDl92+fIkSN2v0mXLp2zatUq3zJ//vlnJyoqytm2bVvIutM+oc+1LV3//fef3Ue0r2jZ+/fv962D9j1/2tba5uGsQ7BffvnFfq79IHjali1b7Pvnn3/erufs2bPtNuvXr5/dHu7+ovUuVaqUc+ONNzrffPON/d7cuXPttkqqbRNqPbQdNE3bJXj7yN69e+2+re+o7t1l/vDDD75l6LjJnj2707dvX7tuemnf1Hx//vmnbz6dF3Tca11DadasmdOoUSO73+u3PvroI+err77ylSNr1qzOQw895GzevNlum3z58vm2WahtrZfWRfu+//bXd1Te1q1bOxs3bnQWLlzoZMqUyR4zjz76qC3/tGnT7Hf99yftM1988YWtx2XLljlXXXWV06NHD9/n7jbs1auXff/EE0/YY/fw4cMB69m2bVt7rgQSgyAWSKEgVmrXru088MADIYPYzp07O926dQv4ri7cClxOnDhh3//777/2gqSLRMGCBZ2RI0cGzB8qEAnFPxh16aKuC+zu3bt903QxUxndwMn9ni7GefLkccaNG+ebV4GDvv/HH38ELLdBgwbOwIED7b914QwObCZOnGjXJS5uADFnzpyA4CxLliw2oPFfroIal4IaTVNZXQcOHLDfmzdvXsD3/C/KCgY0zT8gCaYbgAkTJgTUe5MmTWJdlJs2bZroINYtV65cuWL9rpbnHyAouKhXr16c5dTytVz/bSraR1Rmf0kdxLrfVxDl0n5Qq1Yt3/siRYrE2od1k6agTCZPnmwDUG3vcCV221xIEBtXsKmbQ/8gVjehwSpUqOCMGTPG9/72228PuFENVqlSJWfo0KEhP3vyySdt0Hj+/PmA40nBqG5K49rW/tvRpe2sgDgmJsY3TQGsAk53WaLf0w1BXN59913nsssuC5immw7dnAwaNMjJkCGDPa8Fe/zxx+Pdl4FQaE4ApCC1i33zzTfN5s2bY32mx/t6LK7HeO5LjyD1yHXHjh12Hj1S1GPHV1991ZQuXdo+OkyI//IefPDBOOdTmfQ4UC+XHs3rMb1/efVoslGjRmbw4MGmT58+vukbNmywjx7Lli0b8Jt63KnHvi49flXZXXoEun///gTX47rrrvP9W4+Wr7rqqoByqV2pHlv6r4/a//k/0r7ssstifU/z6DG0S80z/NdZj0bVRELNHzRd66TPVA9xlc99H2o7X4yuXbvax9YnT560j2Xfeecd+7g2LidOnLD/VzOJSOjQoYMtoyh2Utk1TWJiYsyff/5prr/++oDv6L1bb3rEr0fP2t6hRGrbaD9/5pln7GN0lU2/qyYywb9bvXr1WN/t0qWLmT59uv23mup8+umn8W5DNSkaMWKErRd1flq/fr3vM62D1sW/6ZDmU71cSG9/NS3IkSOH772aaegckC5duoBp/sermm80aNDANq3Qd++9917b3EJNnFwqo7aT6kznjBtuuCHWb6sJlf93gHBcei28gVTspptusoHpwIEDbXs1f7rwdO/e3V60gqmNmuvrr7826dOnt+1I1V7W/6ITin9bP7VzvVj58+e37dcUkOji6y5T5Ve5fvrpJ/t/f7rIuzJmzBjwmS7A/58cuji6CCZHxyhdfJcsWWLGjRtnypQpY3+ndevWEemoo7aRavuotoMK2tWeUmWJi9piitoGa7vFR4FK8HbQ8i+G2o3279/f/PzzzzagVlvjtm3bhv191XVybxs3QPNf94TWe+zYsXbEB7UhdtuDqu1r8O9qejC1BdXN58qVK813331nSpYsadvBx0VBr84ZahO+ePFi265+/Pjxtq1rUgt1bIaa5naaVHtitfHt0aOHbSurgF7t+jt37mzrQjesovlXrFhhzwtbt24N+dvq1JXQPgoEIxMLpDANtfXRRx/Zi5i/a665xmzatMlejINfClhEFz1lc/V9BYbqOOJP8ylL5M9/OeqYEdd8ymYpyPDv1KTyqJOLsjEuBQoarsftBKVOM6KMmZapLE1w+dUp5GJ9//33vn8rKPv9999tmeOiz9TBxe0EJMoQ/fbbbwHro3lWr17te6/Ptc7usnXx1Q2HOsIoYNG66OIdX/nc9/GVLz6hto+bNe7YsaPN5OnVrl27eAM9Zbx1k6HtmBAFEP4d7JQpdZ8AXCh12Kpbt67tea6XMvjuPqhy6WZI9etP793to8y6bsLi6rWeFNvGDZz81z2hsVT1u+pQds8999iOaurcqP0xHHoacMcdd9jtpycv6sSXED0d0VOU+fPn20zmlClT7HStg84j/gG4yqYbW9V9XBSYhtq/Eks3rApQFVSrs6Wewii7Hiro//XXX+1TGXXcczPR/tRRU+cQIDEIYoEUpoutHqm+/PLLAdOVsVKQqsBUF1H1ov/www99gaqCRT2qU6ZWvewVFMydO9e89957AY8Dlan9448/7CDicdF8ClD0O5pPPYc1goJbNmXOVq1aZbNGCkLUIzw4w6TMkIIqlUVZWF3A9F19RxdbLV/LUOZI814s9WJftmyZvdgpcFGWUcFAXDTagAINPYJXdkjNNRR06LGnpvtf0JXVUrCri7KWrQuyeny7y9H6qK60jPbt24ccvkvBg3pxK5hR7/d3333XPPbYYxe0rto+qlOtr7aP/2NWZea++OILGwzE9xjazTJqu2r9E3LzzTebmTNnmm+++cY2DVGwHJxRvxDaJzRCgurDbUrg6tu3r70p036smwdlKFXPbr0pk6vAVNtZ9bt9+3bz/vvv+24Ak2Lb6CZA21s3l3o8r0Dr6aefjned9LvKAOt41Xf0BCW+UTyCaRu6zYpUz/FRhldNFXQ86bj88ssvfQG4RgLQTaf2XwWJOl+oyUHv3r0DmgCE2r+0b2kM4YsZ9k83qMpaT5gwwW4b7T+vvfZawDxr1qyxTY801JuaOmg0A9W95ndp/9axd8stt1xwWZBGhWwpCyBZOna51JlEPX+DD0F1oFJPZHXMUI/lypUr+zq+3H///baTx8mTJ33zjx8/3smbN6/tpSwrV66039GIAPEd3lpGq1atnNy5c/t6Kruds9SjX7+tDjUaDeGvv/6Ks0OYeh6rJ/9NN93kHD161Pb6Hzx4sO0Moo4chQsXdu68807bszquDkvBHdyCuZ1q1CtbnXZUbxpBQKMjuOLqCKWOcPfee6/9TB261FHF7fnu/73333/f9qBWvWnkBP/e7dpW9evXt98vVqyY88orr9hOO4899phvHnWcGTZsmK0vdY7RqA0vvfRSQFkS07FLHnzwQdtBRtP9O1eJeuurLsKxaNEip2jRogGdc0J19lFvcXV40mgBWs8ZM2ZcdMcu0TqpXlUvwT3wVSZ1WlL5tL/o9z799NOAeXbu3Gn3VZVLy9BIEm6nu6TaNps2bXKuu+46uxyN0KFOjvF17FJHMx3TOk4LFChgR0W47777Ao7z4HL4U0cslevWW291EvLII484pUuXtnWYP39+uz+rg6Jr+fLltjOcjgutW//+/e3oJfFta408oFE61MnK/SxUZ89Q567g9dIIEzrO3ePrrbfe8tWVOqSqI1twh1WdY3TeOHv2rH3/zjvv2A5jQGJF6T+RDqQBIC4aq7N+/fo2Y5RUf8XKy3TKViZQWThl3MKZX53bNH6wMpuIPGXZ9URAj9Vbtmxp0jplwvWESZl0IDFoTgAAHvHPP/+YV155xT4GDqctpdsR5/XXX/cNgI/IUVMHtRlXL33dkDVv3tykdWouo0CeGyxcCEYnAACPUKcotQVWUKrh1sKlv8SkFyJLQ3BpNAJ1ulKnrkvxT8CmNO3P/fr1i3Qx4FE0JwAAAIDn0JwAAAAAnkMQCwAAAM8hiAUAAIDnEMQCAADAcwhiAQAA4DkEsQAAAPAcglgAAAB4DkEsAAAAPIcgFgAAAMZr/g9zz6HTsivsPwAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "first_candidates = candidates_df[candidates_df['step'] == 1]\n", "fig, ax = plt.subplots(figsize=(7, 3))\n", "ax.barh(first_candidates['piece'][::-1], first_candidates['probability'][::-1])\n", "ax.set(xlabel='Next-token probability (full vocabulary softmax)',\n", " title='First generated token: five leading candidates')\n", "fig.tight_layout()\n", "plt.show()\n", "# Only the leading five are drawn; their probabilities need not sum to one.\n", "# Inspection uses float64 softmax to reduce summation error over the large vocabulary." ] }, { "cell_type": "markdown", "id": "aac990ac", "metadata": {}, "source": [ "Now generate a complete answer. We supply a fresh `GenerationConfig` so that sampling controls\n", "in the model's default configuration cannot silently alter this comparison. The token limit\n", "is a safety bound on output length, not a request to produce exactly that many tokens.\n", "Timing below measures one local generation call, including prompt processing; it is not TTFT,\n", "an isolated decode benchmark, or a comparison against a hosted production service." ] }, { "cell_type": "code", "execution_count": 16, "id": "c8fa8041", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The delivery was scheduled for 18:00, but the driver reports a van breakdown, so the revised arrival time is 18:25.\n", "Stop: eos\n", "Manual generation matches the built-in greedy prefix.\n" ] } ], "source": [ "def generate_answer(messages, do_sample=False, temperature=1.0, seed=7):\n", " encoded = tokenizer.apply_chat_template(\n", " messages, tokenize=True, add_generation_prompt=True,\n", " return_dict=True, return_tensors='pt',\n", " ).to(DEVICE)\n", " sampling_options = {'temperature': temperature, 'top_k': 0, 'top_p': 1.0} if do_sample else {}\n", " settings = GenerationConfig(\n", " max_new_tokens=MAX_NEW_TOKENS,\n", " do_sample=do_sample, num_beams=1,\n", " repetition_penalty=1.0,\n", " eos_token_id=list(EOS_IDS), pad_token_id=PAD_ID,\n", " use_cache=True, **sampling_options,\n", " )\n", " torch.manual_seed(seed)\n", " started = time.perf_counter()\n", " with torch.inference_mode():\n", " result = model.generate(**encoded, generation_config=settings) # runs all the steps till stop token\n", " elapsed = time.perf_counter() - started\n", " \n", " new_ids = result[0, encoded['input_ids'].shape[1]:].tolist()\n", " return {\n", " 'method': 'sampling' if do_sample else 'greedy',\n", " 'temperature': temperature if do_sample else None,\n", " 'seed': seed if do_sample else None,\n", " 'input_tokens': encoded['input_ids'].shape[1],\n", " 'output_tokens': len(new_ids), 'elapsed_seconds': elapsed,\n", " 'stop_reason': 'eos' if new_ids and new_ids[-1] in EOS_IDS else 'token_limit',\n", " 'answer': tokenizer.decode(new_ids, skip_special_tokens=True), # <---- generating text from sequence of token IDs\n", " 'generated_ids': new_ids,\n", " }\n", "\n", "greedy = generate_answer(MESSAGES)\n", "print(greedy['answer'])\n", "print('Stop:', greedy['stop_reason'])\n", "assert greedy['generated_ids'][:len(selected_ids)] == selected_ids\n", "print('Manual generation matches the built-in greedy prefix.')" ] }, { "cell_type": "markdown", "id": "93529b99", "metadata": {}, "source": [ "### 5.1 Change selection, keeping the model and prompt fixed\n", "\n", "First compare the same first-token logits at two temperatures. This isolates the mathematical\n", "effect without conflating it with different later contexts. Then generate two full sampled answers\n", "at each temperature. Two seeds are illustrative, not an evaluation of quality or a statistical study.\n", "Fixed seeds aid reproduction in a fixed environment; exact results can vary by hardware and software." ] }, { "cell_type": "code", "execution_count": 17, "id": "1b18c62a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
temperaturerankpieceprobability
00.41'The'0.831754
10.42'Order'0.058638
20.43'A'0.047344
30.44'Please'0.020882
40.45'Delivery'0.016458
51.01'The'0.298447
61.02'Order'0.103310
71.03'A'0.094836
81.04'Please'0.068357
91.05'Delivery'0.062146
\n", "
" ], "text/plain": [ " temperature rank piece probability\n", "0 0.4 1 'The' 0.831754\n", "1 0.4 2 'Order' 0.058638\n", "2 0.4 3 'A' 0.047344\n", "3 0.4 4 'Please' 0.020882\n", "4 0.4 5 'Delivery' 0.016458\n", "5 1.0 1 'The' 0.298447\n", "6 1.0 2 'Order' 0.103310\n", "7 1.0 3 'A' 0.094836\n", "8 1.0 4 'Please' 0.068357\n", "9 1.0 5 'Delivery' 0.062146" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
methodtemperatureseedinput_tokensoutput_tokenselapsed_secondsstop_reasonanswer
0greedyNaNNaN68341.851978eosThe delivery was scheduled for 18:00, but the driver reports a van breakdown, so the revised arrival time is 18:25.
1sampling0.47.068251.422917eosThe driver's van breakdown is causing an 18:00 to 18:25 delivery delay.
2sampling0.419.068140.607951eosThe delivery was delayed by one hour due to a van breakdown.
3sampling1.07.068401.618705eosDelivery update: The original order for 482 has been updated to be at 18:25 when expected to be picked up. Tank damage is confirmed to be an issue.
4sampling1.019.068401.589425eosOrder 482 is scheduled to arrive at 18:00, but the driver reports a van breakdown, so the pickup time will be 18:25 instead.
\n", "
" ], "text/plain": [ " method temperature seed input_tokens output_tokens elapsed_seconds \\\n", "0 greedy NaN NaN 68 34 1.851978 \n", "1 sampling 0.4 7.0 68 25 1.422917 \n", "2 sampling 0.4 19.0 68 14 0.607951 \n", "3 sampling 1.0 7.0 68 40 1.618705 \n", "4 sampling 1.0 19.0 68 40 1.589425 \n", "\n", " stop_reason \\\n", "0 eos \n", "1 eos \n", "2 eos \n", "3 eos \n", "4 eos \n", "\n", " answer \n", "0 The delivery was scheduled for 18:00, but the driver reports a van breakdown, so the revised arrival time is 18:25. \n", "1 The driver's van breakdown is causing an 18:00 to 18:25 delivery delay. \n", "2 The delivery was delayed by one hour due to a van breakdown. \n", "3 Delivery update: The original order for 482 has been updated to be at 18:25 when expected to be picked up. Tank damage is confirmed to be an issue. \n", "4 Order 482 is scheduled to arrive at 18:00, but the driver reports a van breakdown, so the pickup time will be 18:25 instead. " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "temperature_rows = []\n", "for temperature in (0.4, 1.0):\n", " probabilities = torch.softmax(first_logits.double() / temperature, dim=-1)\n", " values, candidates = probabilities.topk(5)\n", " for rank, (candidate, probability) in enumerate(zip(candidates.tolist(), values.tolist()), 1):\n", " temperature_rows.append({\n", " 'temperature': temperature, 'rank': rank,\n", " 'piece': repr(tokenizer.decode([candidate])), 'probability': probability,\n", " })\n", "display(pd.DataFrame(temperature_rows))\n", "\n", "runs = [greedy] # the previous greedy run\n", "for temperature in (0.4, 1.0):\n", " for seed in SEEDS:\n", " runs.append(generate_answer(MESSAGES, do_sample=True, temperature=temperature, seed=seed))\n", "run_table = pd.DataFrame([{k: v for k, v in row.items() if k != 'generated_ids'} for row in runs])\n", "with pd.option_context('display.max_colwidth', 240):\n", " display(run_table)" ] }, { "cell_type": "code", "execution_count": 22, "id": "dad85eb0-787b-452b-aaad-27685adbb02e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "5\n", "\n", "{'method': 'sampling', 'temperature': 1.0, 'seed': 19, 'input_tokens': 68, 'output_tokens': 40, 'elapsed_seconds': 1.5894245409999712, 'stop_reason': 'eos', 'answer': 'Order 482 is scheduled to arrive at 18:00, but the driver reports a van breakdown, so the pickup time will be 18:25 instead.', 'generated_ids': [4431, 220, 19, 23, 17, 374, 13537, 311, 17331, 518, 220, 16, 23, 25, 15, 15, 11, 714, 279, 5579, 6682, 264, 5242, 29985, 11, 773, 279, 29586, 882, 686, 387, 220, 16, 23, 25, 17, 20, 4518, 13, 151645]}\n" ] } ], "source": [ "print(len(runs))\n", "print(type(runs[0]))\n", "print(runs[-1])" ] }, { "cell_type": "markdown", "id": "6aa9265b", "metadata": {}, "source": [ "## 6. Where the model ends and the system begins\n", "\n", "The model generated text. An application must still decide whether to use it and send it.\n", "Here the complete minimal path is:\n", "\n", "**Synthetic delivery data → construct prompt → model → review → notification service.**\n", "\n", "The final notification service is deliberately represented by a local preview. No message is sent.\n", "An application could add retrieval (RAG), persistent memory, or tools; none is required for this\n", "example. We do not implement them in this introductory notebook.\n", "\n", "The checks below are deliberately limited: preserving two strings does not establish factual\n", "accuracy, a correct cause, appropriate tone, exactly one sentence, or readiness for customers.\n", "Do not treat these checks as a production guardrail or a general evaluation benchmark." ] }, { "cell_type": "code", "execution_count": 23, "id": "e55f1d25", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
methodtemperatureseedcontains_order_idcontains_revised_timehit_token_limitanswer_for_human_review
0greedyNaNNaNFalseTrueFalseThe delivery was scheduled for 18:00, but the driver reports a van breakdown, so the revised arrival time is 18:25.
1sampling0.47.0FalseTrueFalseThe driver's van breakdown is causing an 18:00 to 18:25 delivery delay.
2sampling0.419.0FalseFalseFalseThe delivery was delayed by one hour due to a van breakdown.
3sampling1.07.0TrueTrueFalseDelivery update: The original order for 482 has been updated to be at 18:25 when expected to be picked up. Tank damage is confirmed to be an issue.
4sampling1.019.0TrueTrueFalseOrder 482 is scheduled to arrive at 18:00, but the driver reports a van breakdown, so the pickup time will be 18:25 instead.
\n", "
" ], "text/plain": [ " method temperature seed contains_order_id contains_revised_time \\\n", "0 greedy NaN NaN False True \n", "1 sampling 0.4 7.0 False True \n", "2 sampling 0.4 19.0 False False \n", "3 sampling 1.0 7.0 True True \n", "4 sampling 1.0 19.0 True True \n", "\n", " hit_token_limit \\\n", "0 False \n", "1 False \n", "2 False \n", "3 False \n", "4 False \n", "\n", " answer_for_human_review \n", "0 The delivery was scheduled for 18:00, but the driver reports a van breakdown, so the revised arrival time is 18:25. \n", "1 The driver's van breakdown is causing an 18:00 to 18:25 delivery delay. \n", "2 The delivery was delayed by one hour due to a van breakdown. \n", "3 Delivery update: The original order for 482 has been updated to be at 18:25 when expected to be picked up. Tank damage is confirmed to be an issue. \n", "4 Order 482 is scheduled to arrive at 18:00, but the driver reports a van breakdown, so the pickup time will be 18:25 instead. " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "{'delivery_channel': 'preview_only',\n", " 'sent': False,\n", " 'human_review_required': True,\n", " 'proposed_message': 'The delivery was scheduled for 18:00, but the driver reports a van breakdown, so the revised arrival time is 18:25.'}" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "review_rows = []\n", "for run in runs:\n", " answer = run['answer']\n", " review_rows.append({\n", " 'method': run['method'], 'temperature': run['temperature'], 'seed': run['seed'],\n", " 'contains_order_id': '482' in answer,\n", " 'contains_revised_time': '18:25' in answer,\n", " 'hit_token_limit': run['stop_reason'] == 'token_limit',\n", " 'answer_for_human_review': answer,\n", " })\n", "review_table = pd.DataFrame(review_rows)\n", "with pd.option_context('display.max_colwidth', 240):\n", " display(review_table)\n", "\n", "notification_preview = {\n", " 'delivery_channel': 'preview_only',\n", " 'sent': False,\n", " 'human_review_required': True,\n", " 'proposed_message': greedy['answer'],\n", "}\n", "display(notification_preview)\n", "assert notification_preview['sent'] is False" ] }, { "cell_type": "markdown", "id": "04597eb8", "metadata": {}, "source": [ "Review the actual answers: do they preserve the revised time and van breakdown, make clear that\n", "the delivery is delayed, address the customer, and avoid invented promises or compensation?\n", "If a sample fails, keep it as evidence and discuss it. Do not quietly replace it with a written answer." ] }, { "cell_type": "markdown", "id": "8bf5473b", "metadata": {}, "source": [ "## 7. What this means when you build software\n", "\n", "The prompt supplies the facts for this interaction. Training shaped the model before this request.\n", "Tokenization and the chat template define its input. The model computes representations and scores;\n", "the generation procedure selects a continuation. Your application owns what happens to that continuation.\n", "\n", "The checks below establish mechanical properties of this walkthrough. They do not establish\n", "general model reliability, verify all numerical weights, or benchmark production performance." ] }, { "cell_type": "code", "execution_count": 26, "id": "2e5d64d3", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
checkpassed
0prompt_round_tripTrue
1chat_template_ids_matchTrue
2probabilities_normalisedTrue
3manual_prefix_matches_generateTrue
4all_runs_within_output_boundTrue
5no_notification_sentTrue
\n", "
" ], "text/plain": [ " check passed\n", "0 prompt_round_trip True\n", "1 chat_template_ids_match True\n", "2 probabilities_normalised True\n", "3 manual_prefix_matches_generate True\n", "4 all_runs_within_output_bound True\n", "5 no_notification_sent True" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"model_id\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n", " \"revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n", " \"python\": \"3.9.6\",\n", " \"platform\": \"macOS-26.6.2-arm64-arm-64bit\",\n", " \"device\": \"cpu\",\n", " \"dtype\": \"torch.float32\",\n", " \"torch_threads\": 2,\n", " \"versions\": {\n", " \"torch\": \"2.8.0\",\n", " \"transformers\": \"4.57.6\",\n", " \"tokenizers\": \"0.22.2\",\n", " \"huggingface-hub\": \"0.36.2\",\n", " \"safetensors\": \"0.7.0\",\n", " \"pandas\": \"2.3.3\",\n", " \"matplotlib\": \"3.9.4\",\n", " \"nbformat\": \"5.10.4\",\n", " \"ipykernel\": \"6.31.0\"\n", " },\n", " \"started_at_utc\": \"2026-11-09T23:52:16.972751+00:00\",\n", " \"max_new_tokens\": 64,\n", " \"trace_steps\": 8,\n", " \"sampling_temperatures\": [\n", " 0.4,\n", " 1.0\n", " ],\n", " \"seeds\": [\n", " 7,\n", " 19\n", " ],\n", " \"sampling_top_k\": 0,\n", " \"sampling_top_p\": 1.0,\n", " \"attention_implementation\": \"eager\",\n", " \"verification\": {\n", " \"prompt_round_trip\": true,\n", " \"chat_template_ids_match\": true,\n", " \"probabilities_normalised\": true,\n", " \"manual_prefix_matches_generate\": true,\n", " \"all_runs_within_output_bound\": true,\n", " \"no_notification_sent\": true\n", " },\n", " \"finished_at_utc\": \"2026-11-10T00:35:41.583526+00:00\"\n", "}\n" ] } ], "source": [ "verification = {\n", " 'prompt_round_trip': tokenizer.decode(raw_ids) == PROMPT, \n", " 'chat_template_ids_match': tokenizer.encode(formatted_prompt, add_special_tokens=False) == input_ids[0].tolist(),\n", " 'probabilities_normalised': all(abs(total - 1.0) < 1e-5 for total in probability_sums),\n", " 'manual_prefix_matches_generate': greedy['generated_ids'][:len(selected_ids)] == selected_ids,\n", " 'all_runs_within_output_bound': all(0 < run['output_tokens'] <= MAX_NEW_TOKENS for run in runs),\n", " 'no_notification_sent': notification_preview['sent'] is False,\n", "}\n", "assert all(verification.values()), verification\n", "manifest.update({\n", " 'max_new_tokens': MAX_NEW_TOKENS, 'trace_steps': TRACE_STEPS,\n", " 'sampling_temperatures': [0.4, 1.0], 'seeds': SEEDS,\n", " 'sampling_top_k': 0, 'sampling_top_p': 1.0,\n", " 'attention_implementation': 'eager',\n", " 'prompt': PROMPT, 'formatted_prompt': formatted_prompt,\n", " 'verification': verification,\n", " 'finished_at_utc': datetime.now(timezone.utc).isoformat(),\n", "})\n", "display(pd.DataFrame(verification.items(), columns=['check', 'passed']))\n", "print(json.dumps({k: v for k, v in manifest.items() if k not in ('prompt', 'formatted_prompt')}, indent=2))" ] }, { "cell_type": "markdown", "id": "1432357d", "metadata": {}, "source": [ "### 7.1 Optional W&B record — offline only\n", "\n", "Set `LOG_WANDB = True` in Section 2 to record the measured tables and manifest locally.\n", "The mode is explicitly `offline`: this cell does not log in or upload. It stores synthetic prompts,\n", "responses, and run metadata under the ignored `wandb/` directory. The earlier tables and chart\n", "work without W&B. Uploading or sharing a run is a later decision. Example output:\n", "```\n", "W&B syncing is set to `offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.\n", "Run data is saved locally in ./wandb/offline-run-\n", "Saved an offline W&B record. No upload was requested.\n", "You can sync this run to the cloud by running:\n", "wandb sync ./wandb/offline-run-\n", "Find logs at: ./wandb/offline-run-/logs\n", "```\n", "\n", "\n", "[W&B Tables documentation](https://docs.wandb.ai/models/track/log/log-tables)" ] }, { "cell_type": "code", "execution_count": 28, "id": "0920affe", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "W&B logging skipped. All demonstrations and checks above ran without it.\n" ] } ], "source": [ "if LOG_WANDB:\n", " import wandb\n", " local_wandb = Path('wandb')\n", " local_wandb.mkdir(exist_ok=True)\n", " with wandb.init(\n", " project='bts-what-is-an-llm', mode='offline', dir=str(Path.cwd()),\n", " config=manifest, save_code=False,\n", " settings=wandb.Settings(disable_git=True),\n", " ) as run:\n", " run.log({\n", " 'generation_runs': wandb.Table(dataframe=run_table),\n", " 'greedy_steps': wandb.Table(dataframe=steps_df),\n", " 'next_token_candidates': wandb.Table(dataframe=candidates_df),\n", " 'limited_review_checks': wandb.Table(dataframe=review_table),\n", " })\n", " print('Saved an offline W&B record. No upload was requested.')\n", "else:\n", " print('W&B logging skipped. All demonstrations and checks above ran without it.')" ] }, { "cell_type": "markdown", "id": "4df612d8", "metadata": {}, "source": [ "### 7.2 Comprehension checks and next reading\n", "\n", "1. Why is the complete input longer than the typed prompt? Point to the actual added tokens.\n", "2. Why can the highest-probability next token still lead to a factually wrong answer?\n", "3. Which component would need to fetch a current ETA or send a real notification?\n", "\n", "Optional experiments: change only “to notify the customer” to “for an internal operations log”;\n", "or change only the revised time and inspect whether the output preserves it. Re-run from Section 1\n", "to keep all dependent tables consistent. Do not add private delivery data.\n", "\n", "Follow-up topics: a deeper tokenizer walkthrough; attention and transformers; context and retrieval;\n", "and evaluating generated answers. This notebook deliberately does not implement training, RAG,\n", "tool calling, persistent memory, or production serving optimisations." ] }, { "cell_type": "code", "execution_count": null, "id": "eec7d995-01fc-4c20-ab8d-89347c814200", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" } }, "nbformat": 4, "nbformat_minor": 5 }