Models are stored in 32, 16 or 8 bits. We checked with PyTorch what each format does with the same numbers: fp16 turns 70,000 into infinity, bf16 cannot tell 1.001 from 1, and adding 0.001 ten thousand times stops at 4 in fp16 and at 0.5 in bf16. One outlier made int8 error 12 times worse.
A modelâs weights are just numbers, and how many bits each number gets decides how much memory the model needs. For the 494 million parameters of Qwen2.5-0.5B:
Python integers have no fixed width, so ~5 is -6 and 152 ÂŤÂ 2 is 608. An 8-bit visualizer shows what the same operations do inside a fixed word: twoâs complement, logical and arithmetic shifts, x & (x - 1), x & -x and the XOR trick, each checked against plain Python.
Greedy decoding picks the most likely token at every step. We compared it with beam search on a local Qwen2.5-0.5B for 10 prompts: beam search found a more probable continuation in 8 of 10, and on one prompt greedyâs locally best word led to a wrong answer. Higher probability was not always better text, either.
When do_sample=False, a language model uses greedy decoding: at every step it takes the single most likely next token and never looks back. That is a greedy algorithm in the exact sense of our greedy algorithms post, and it fails the same way: the best-looking step now can lead to a worse result overall.
Three greedy rules run next to an exact answer in a small pygame app. Greedy change-making used 3 coins where 2 were enough, âearliest startâ booked 1 meeting where 3 fit, and value-per-weight packing got 32 where 44 was possible. The rules that do work are just as concrete.
A greedy algorithm makes the choice that looks best right now and never takes it back. It is fast and easy to write. Sometimes it is also the optimal answer, and sometimes it is not, and the difference is not obvious from the code.
An LLM generates one token at a time, and without a cache it recomputes the whole sequence at every step. We generated the same text with a local Qwen2.5-0.5B with the cache on and off: identical tokens, 4.5x faster at 256 tokens. The price is memory, which we calculate for the same model.
An LLM writes one token at a time. To choose token 100, it needs information from tokens 1 to 99. Without a cache, the model reads all 99 again at every step and recomputes the same numbers it computed one step earlier. The KV cache keeps those numbers and reuses them. It is the same trick as memoizing Fibonacci in our dynamic programming post: compute each piece once, look it up afterwards.
Three classic dynamic programming problems drawn cell by cell in a small pygame app. Memoization cut Fibonacci(8) from 67 calls to 15, the LCS table picked a different answer when the strings were swapped, and the knapsack table found 44 where a value-per-weight greedy got 32.
Dynamic programming (DP) has one idea: when a problem breaks into smaller problems that repeat, solve each one once and reuse the answer. The repetition is easy to say and hard to picture, so this post draws it. The three lessons come from the same pygame app as the path search post, and every number below came out of an actual run:
We embedded 134k docstring paragraphs with a local model and compared exact nearest-neighbour search with an HNSW index. Exact search grew from 0.07 to 0.8 ms per query; HNSW stayed near 0.17 ms at 97% recall. And with only one candidate, HNSWâs greedy walk found the true nearest neighbour less than half the time.
Semantic search turns every text into a vector and returns the vectors closest to the query (embeddings explained). The simplest way to find them is to compare the query with every vector. Vector databases use an approximate index instead, most often HNSW (Hierarchical Navigable Small World). The question is when the approximation is worth it.
Four grid searches run in a small pygame app on fixed maps. DFS found a 25-step path where BFS found 13, BFS walked through a swamp that Dijkstra avoided, and A* reached the same optimal cost as Dijkstra after expanding 22 cells instead of 267.
All four algorithms âfind a pathâ. They do not promise the same path. On four fixed maps in a small pygame app, measured with the appâs own search code:
TabPFN-3.5âs reported benchmark lead is a reason to evaluate, not a universal replacement rule. A local LightGBM demo scores 1.0000 AUC on random rows and 0.4748 on unseen groups; no TabPFN model is run.
A perfect test score can disappear before you change models. In our constructed example, one LightGBM baseline scored 1.0000 ROC AUC with random-row splitting and 0.4748 when testing unseen groups. The change was the evaluation question, not a new foundation model.
Language models read tokens, not letters: â strawberryâ is a single token. We asked two small local models to count letters in three words, with and without spelling them out first. The answers were inconsistent; code was not.
A language model never sees the letters of âstrawberryâ. It sees tokens, and inside a sentence the whole word ` strawberry` (with its leading space) is one token. To count the râs, the model has to recall how that token is spelled, then count, all from patterns it learned. Sometimes that works. In our test it was not reliable, and spelling the word out first did not fix it. If you need a letter count, count with code.
