Greedy decoding picks the most likely token at every step. We compared it with beam search on a local Qwen2.5-0.5B for 10 prompts: beam search found a more probable continuation in 8 of 10, and on one prompt greedyâs locally best word led to a wrong answer. Higher probability was not always better text, either.
When do_sample=False, a language model uses greedy decoding: at every step it takes the single most likely next token and never looks back. That is a greedy algorithm in the exact sense of our greedy algorithms post, and it fails the same way: the best-looking step now can lead to a worse result overall.
Three greedy rules run next to an exact answer in a small pygame app. Greedy change-making used 3 coins where 2 were enough, âearliest startâ booked 1 meeting where 3 fit, and value-per-weight packing got 32 where 44 was possible. The rules that do work are just as concrete.
A greedy algorithm makes the choice that looks best right now and never takes it back. It is fast and easy to write. Sometimes it is also the optimal answer, and sometimes it is not, and the difference is not obvious from the code.
An LLM generates one token at a time, and without a cache it recomputes the whole sequence at every step. We generated the same text with a local Qwen2.5-0.5B with the cache on and off: identical tokens, 4.5x faster at 256 tokens. The price is memory, which we calculate for the same model.
An LLM writes one token at a time. To choose token 100, it needs information from tokens 1 to 99. Without a cache, the model reads all 99 again at every step and recomputes the same numbers it computed one step earlier. The KV cache keeps those numbers and reuses them. It is the same trick as memoizing Fibonacci in our dynamic programming post: compute each piece once, look it up afterwards.
Three classic dynamic programming problems drawn cell by cell in a small pygame app. Memoization cut Fibonacci(8) from 67 calls to 15, the LCS table picked a different answer when the strings were swapped, and the knapsack table found 44 where a value-per-weight greedy got 32.
Dynamic programming (DP) has one idea: when a problem breaks into smaller problems that repeat, solve each one once and reuse the answer. The repetition is easy to say and hard to picture, so this post draws it. The three lessons come from the same pygame app as the path search post, and every number below came out of an actual run:
We embedded 134k docstring paragraphs with a local model and compared exact nearest-neighbour search with an HNSW index. Exact search grew from 0.07 to 0.8 ms per query; HNSW stayed near 0.17 ms at 97% recall. And with only one candidate, HNSWâs greedy walk found the true nearest neighbour less than half the time.
Semantic search turns every text into a vector and returns the vectors closest to the query (embeddings explained). The simplest way to find them is to compare the query with every vector. Vector databases use an approximate index instead, most often HNSW (Hierarchical Navigable Small World). The question is when the approximation is worth it.
Four grid searches run in a small pygame app on fixed maps. DFS found a 25-step path where BFS found 13, BFS walked through a swamp that Dijkstra avoided, and A* reached the same optimal cost as Dijkstra after expanding 22 cells instead of 267.
All four algorithms âfind a pathâ. They do not promise the same path. On four fixed maps in a small pygame app, measured with the appâs own search code:
TabPFN-3.5âs reported benchmark lead is a reason to evaluate, not a universal replacement rule. A local LightGBM demo scores 1.0000 AUC on random rows and 0.4748 on unseen groups; no TabPFN model is run.
A perfect test score can disappear before you change models. In our constructed example, one LightGBM baseline scored 1.0000 ROC AUC with random-row splitting and 0.4748 when testing unseen groups. The change was the evaluation question, not a new foundation model.
Language models read tokens, not letters: â strawberryâ is a single token. We asked two small local models to count letters in three words, with and without spelling them out first. The answers were inconsistent; code was not.
A language model never sees the letters of âstrawberryâ. It sees tokens, and inside a sentence the whole word ` strawberry` (with its leading space) is one token. To count the râs, the model has to recall how that token is spelled, then count, all from patterns it learned. Sometimes that works. In our test it was not reliable, and spelling the word out first did not fix it. If you need a letter count, count with code.
A synthetic task-selection example flips a 32.5% time reduction into a 10% increase without changing any task. Use it to understand why METRâs follow-up is not a simple verdict on AI coding productivity.
You can get âAI made work fasterâ and âAI made work slowerâ from the same task list without changing a single completion time. Change which tasks enter the study, and you change the question the result answers.
We rounded a small modelâs weights to 8, 4, 3 and 2 bits and measured perplexity and correct answers. 8-bit changed nothing; 4-bit dropped the 0.5B model from 11 to 4 correct answers, while a 1.5B model kept all 12.
TL;DR â Quantization stores a modelâs weights with fewer bits, so the model takes less memory. In our test, 8-bit weights were as good as the original. At 4 bits, a 0.5B model answered only 4 of 12 handbook questions correctly (down from 11), while a 1.5B model still got all 12. Below 4 bits, the small model broke down completely.
