How LLMs Actually Predict Words
A common misconception is that Large Language Models (LLMs) "think" like humans or store pre-computed answers to every query. In reality, modern foundation models are probabilistic next-token prediction engines.
1. What is a Token?
Language models do not process text letter by letter or word by word. Instead, they operate on tokens—chunks of characters, syllables, or common sub-words.
- In English, 1 token ≈ 4 characters or ~0.75 words.
- For example, the sentence:
"Learning AI is empowering."is broken down by Byte-Pair Encoding (BPE) into tokens like:["Learn", "ing", " AI", " is", " empower", "ing", "."]
# Conceptual Tokenization Illustration
tokens = tokenizer.encode("Learning AI with AI Studio")
print(tokens) # [32541, 9552, 449, 9552, 6109]
2. The Probability Distribution
When you submit a prompt, the model calculates a softmax probability distribution over its entire vocabulary (typically 32,000 to 128,000 tokens) for what token is most statistically coherent next.
$$\text{Probability}(w_t \mid w_1, w_2, \dots, w_{t-1}) = \frac{\exp(z_t / T)}{\sum_j \exp(z_j / T)}$$
Where $T$ is the temperature parameter:
- Temperature = 0.0: Greedy decoding. The model deterministically selects the single highest-probability token. Ideal for code generation, math, and factual data extraction.
- Temperature = 0.7 - 0.9: High variety and creative sampling. Allows lower-probability tokens to be picked, generating poetic, unexpected, or brainstormed variations.
3. Why Prompt Clarity Matters
Because the model predicts based on the preceding context window, every word you supply reshapes the probability landscape. If your prompt is ambiguous:
"Write about cars."
The model's probability distribution is flat and unfocused. But when you supply constraints, tone, and goals:
"Act as an automotive mechanical engineer. Summarize the thermodynamic efficiency trade-offs between electric vehicle battery pre-conditioning and cabin climate heating in sub-zero winter conditions. Use bullet points."
You collapse the probability space into high-precision, technical tokens.